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Abstract 

Past  research  shows  predicting  cost  growth  is  an  important  topic  with  DoD 
systems.  Researchers  have  attempted  to  predict  total  program  cost  growth  as  well  as 
identify  predictors  of  program  cost  growth.  Our  research  addresses  this  through  examining 
cost  growth  at  reviews  and  milestones  along  an  aircraft’s  schedule.  We  assess  cost  growth 
factors  at  four  major  reviews,  Critical  Design  Review,  First  Flight,  Development  Test  and 
Evaluation  End,  and  Initial  Operating  Capability. 

The  first  portion  of  the  analysis  focuses  on  identifying  cost  growth  factors  and 
percent  of  total  cost  growth  at  the  four  program  reviews.  The  second  portion  identifies 
predictors  of  cost  growth  at  the  four  reviews.  In  our  results,  we  present  a  spike  in 
procurement  cost  growth  first  occurring  around  First  Flight  and  we  identify  the  median 
percent  of  total  cost  growth  at  IOC,  or  48  percent  of  program  completion  to  be  91%.  The 
second  portion  of  the  results  identifies  the  three  most  common  predictors  of  cost  growth  at 
program  reviews:  Bombers,  Prototyping,  and  electronic  aircraft  system  upgrades. 
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PREDICTING  COST  GROWTH  USING  PROGRAM  REVIEWS  AND 


MILESTONES  FOR  DOD  AIRCRAFT 


I.  Introduction 


General  Issue 

Cost  growth  occurring  in  Department  of  Defense  (DoD)  weapon  systems  is  a  major 
problem  for  Congress.  Cost  growth  forces  Congress  to  adjust  DoD  funding  and  shift 
priorities  in  order  to  compensate  for  cost  growth  in  major  weapon  systems.  The  RAND 
Corporation  conducted  a  study  in  1993  which  states  Major  Defense  Acquisition  Programs 
(MDAPs)  historically  experience  cost  growth  of  20  percent  from  the  initial  baseline 
estimate  (Drezner  et  ah,  1993). 

Cost  growth  in  weapon  systems  creates  challenges  for  the  DoD,  Air  Force,  and 
civilian  contractors  involved  in  developing  the  United  States’  most  advanced  weapon 
systems.  At  any  level,  (DoD,  Air  Force,  or  contractor)  cost  growth  challenges  the  parties 
involved  and  often  forces  difficult  decisions  in  regards  to  funding.  In  a  major  DoD 
program,  cost  growth  can  remove  funding  from  smaller  programs,  postpone  program 
development,  or  eliminate  programs.  In  order  to  better  prepare  the  DoD  for  funding  issues 
with  MDAPs,  Program  Managers  (PMs)  must  prepare  for  program  cost  growth. 

Program  Managers  control  the  day-to-day  operations  with  MDAPs.  DoD  programs 
face  several  types  of  risk:  cost,  schedule,  and  perfonnance.  Cost  risk  challenges  all 
government  weapon  systems  and  could  eliminate  one  or  more  programs  from  activation. 
Schedule  risk  causes  program  activation  to  delay,  and  the  delays  can  jeopardize  national 
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defense.  Performance  risk  limits  the  weapon  system’s  ability  to  conduct  the  mission.  If  a 
weapon  system  experiences  perfonnance  or  schedule  setbacks,  cost  risk  is  likely  to  follow. 

No  matter  the  type  of  risk,  most  programs  will  likely  need  additional  funding  to 
combat  the  problems.  To  better  prepare  for  cost  growth,  PMs  need  to  know  when  the  cost 
growth  will  occur.  Identifying  when  growth  cost  is  likely  to  occur  allows  baseline 
estimates  to  reflect  what  could  happen  in  a  weapon  system’s  future.  Without  more 
accurate  cost  estimates,  PMs  and  cost  estimators  need  to  better  prepare  for  the  reality  of 
requesting  additional  risk  dollars  to  support  MDAP  cost  growth. 

Arena  et  al.  (2006)  found  the  average  total  program  cost  growth  for  programs 
similar  to  Air  Force  programs  was  46%.  The  total  program  life  cycle  includes  all  program 
activity:  Research  and  Development,  Procurement,  Operations  and  Sustainment,  and 
Disposal.  Cost  growth  negatively  impacts  DoD  programs  by  tightening  budget  restrictions 
and  minimizing  funding  flexibility  for  DoD  leaders.  For  this  research,  cost  growth  is 
defined  as  the  increase  in  cost  from  the  Development  Estimate  (DE)  to  the  Current  or  final 
Estimate  CE)  of  the  DoD  program.  The  DE  occurs  at  MS  B  when  a  program  officially 
becomes  a  “program  of  record.”  There  are  many  techniques  to  develop  a  cost  estimate. 
The  three  most  common  methods  used  to  develop  cost  estimates  are:  analogy,  engineering 
build-up,  and  parametric.  Each  estimating  method  has  its  advantages  depending  on  the 
degree  of  knowledge  and  the  placement  along  the  acquisition  life  cycle  of  the  program 
under  development.  Chapter  2  further  examines  the  three  cost  estimating  techniques  as 
well  as  several  supplemental  techniques. 
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Problem  Statement 


Instead  of  trying  to  generate  more  accurate  estimates  which  allow  programs  to 
sustain  less  cost  growth,  PMs  could  better  prepare  for  the  event  of  weapon  system  cost 
growth.  As  the  subsequent  chapters  will  explain,  this  research  identifies  how  much  cost 
growth  occurs  at  different  reviews  along  an  aircraft’s  schedule.  In  addition  to  identifying 
the  amount  of  cost  growth  at  different  reviews,  this  research  identifies  significant 
predictors  of  cost  growth  at  program  reviews.  With  the  knowledge  of  how  much  cost 
growth  programs  sustain  at  different  reviews  and  what  variables  are  significant  predictors 
of  cost  growth,  PMs  can  notify  higher  authorities  before  funding  needs  become  a  major 
problem,  and  their  effects  spread  through  DoD  programs. 

There  are  several  techniques  to  measure  where  cost  growth  occurs.  One  way  is  to 
measure  the  percent  complete  of  a  program  when  cost  growth  occurs.  Measuring  cost 
growth  at  percent  complete  can  cause  discrepancies  between  programs  because  the  percent 
complete  can  occur  at  different  stages  of  weapon  system’s  life  cycle.  A  second  technique 
is  to  examine  specific  dates  that  are  consistent  across  MDAPs  and  test  to  see  if  cost  growth 
occurs  at  the  specific  dates. 

All  MDAPs  are  required  to  pass  certain  reviews  and  milestones.  Four  common 
reviews  for  all  major  aircraft  programs  are  Critical  Design  Review  (CDR),  First  Flight 
(FF),  Development  Test  &  Evaluation  End  (DT&E),  and  Initial  Operating  Capability 
(IOC).  The  three  Milestones  for  MDAP  programs  are  Milestone  A,  B,  C.  In  (2000) 
Milestones  A,  B,  C  replaced  Milestones  I,  II,  III.  Further  information  on  the  transition 
from  MS  I,  II,  III,  to  A,  B,  C  is  available  in  Chapter  2.  With  the  collection  of  program 
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reviews,  the  model  has  four  significant  dates  to  test  for  statistical  significance  of  cost 
growth  across  aircraft  programs. 

Research  Objectives  and  Scope 

The  objective  of  this  research  is  to  better  prepare  PMs  for  cost  growth  in  weapon 
systems.  The  way  to  better  prepare  for  cost  growth  is  to  realize  that  cost  growth  is  going  to 
occur  and  plan  for  the  cost  growth  well  in  advance  so  the  weapon  system  and  other  DoD 
programs  are  not  significantly  impacted.  The  objective  of  this  research  is  best  summarized 
by  the  following  objectives: 

1-  Identify  a  significant  review  along  an  Aircraft’s  schedule  where  cost  growth 
occurs. 

2-  Identify  predictors  of  cost  growth  at  program  reviews. 

Graphically  display  trends  of  cost  growth  along  an  aircraft’s  schedule  based 

This  research  focuses  on  major  aircraft  weapon  systems.  Focusing  specifically  on 
aircraft  allows  for  consistent  analysis  on  one  type  of  DoD  platform.  In  theory,  the 
development  of  all  aircraft  should  follow  a  similar  schedule.  Based  on  the  type  of  aircraft 
(Fighter,  Bomber,  Tanker),  some  portions  of  the  program  lifecycle  may  be  more  complex 
than  others.  The  complexity  has  an  effect  on  reviews  and  milestones  in  the  program’s 
schedule.  If  the  research  has  statistically  significant  findings,  the  methods  can  expand  to 
cover  additional  weapon  system  platforms. 

Methodology 

Since  1969,  MDAPs  are  required  to  submit  Selected  Acquisition  Reports  (SARs) 
(Drezner  et  ah,  1993).  SARs  outline  a  weapon  systems  status  and  report  current  funding 
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estimates  as  well  as  actual  expenses  incurred.  SARs  are  required  to  report  annually,  and 
provide  a  common  ground  to  evaluate  weapon  systems.  For  our  research,  we  use  SARs  to 
evaluate  program  estimates  and  actual  costs.  Additional  infonnation  on  SARs  is  available 
in  Chapter  3. 

This  research  uses  several  methods  to  conduct  cost  growth  analysis.  First, 
graphical  analysis  provides  a  method  to  examine  the  cost  growth  of  each  aircraft  weapon 
system.  Graphical  analysis  presents  a  visual  depiction  of  cost  growth  which  can  aid  in 
developing  predictors.  Next,  logistic  regression  identifies  if  cost  growth  occurs  at  program 
reviews  along  an  aircraft’s  schedule.  Finally,  Fisher’s  Exact  Test  and  Odds  Ratios  identify 
possible  predictors  of  cost  growth  and  odds  of  an  event  occurring. 

Preview 

The  following  chapters  discuss  the  Literature  Review,  Methodology,  Analysis, 
Results,  and  Conclusion/Discussion.  The  Literature  Review  discusses  approaches  to  cost 
estimating,  relevant  cost  growth  studies  of  the  past,  and  potential  predictor  variables  of  cost 
growth.  The  Methodology  chapter  discusses  Logistic  Regression,  Fisher’s  Exact  Test, 
Odds  Ratio,  and  how  these  three  methods  work  together  to  identify  predictors  of  cost 
growth.  The  Analysis  and  Results  sections  recaps  the  results  of  the  research  and  provides 
significant  predictors  of  cost  growth  at  different  stages  of  an  aircraft’s  schedule  that  can 
help  mitigate  the  DoD’s  inevitable  cost  growth  problem.  The  Conclusion/Discussion 
addresses  the  goals  of  this  research,  discusses  limitations  of  the  study,  and  provides  some 
thoughts  on  possible  avenues  for  future  research. 
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II.  Literature  Review 


Chapter  Overview 

Chapter  2  provides  an  overview  of  topics  related  to  cost  estimating  and  cost  growth 
in  the  DoD.  First,  we  provide  an  outline  on  the  important  cost  estimating  techniques  to 
provide  background  and  an  understanding  of  available  tools  to  cost  estimators.  Second,  we 
review  historical  studies  of  cost  growth.  Third,  the  literature  review  presents  AFIT  theses 
beginning  in  2002  which  used  logistic  and  multiple  regression  to  predict  cost  growth. 
Lastly,  we  offer  a  review  of  important  predictor  variables  in  estimating  cost  growth. 

Cost  Estimating  Techniques 

To  develop  a  cost  estimate,  the  cost  estimator  should  perform  certain  processes. 
The  estimator  may  develop  the  estimate  using  the  weapon  system  work  breakdown 
structure  (WBS)  and  generate  a  best  estimate  for  each  piece  of  the  WBS.  The  estimate 
must  be  in  constant-year  dollars,  and  include  all  assumptions  in  generating  the  cost  model. 
In  addition,  the  estimate  should  be  time-phased  by  allocating  the  costs  for  parts  of  the 
weapons  system  to  the  years  in  which  the  costs  will  likely  occur.  After  completing  the 
cost,  the  estimator  must  validate  their  work.  Methods  for  validating  include  double 
checking  and  cross  checking  for  errors  or  double  counting,  comparing  the  estimate  against 
other  independent  estimates  for  differences,  and  updating  the  estimates  as  data  becomes 
available  (GAO,  2009).  The  cost  estimate  is  an  iterative  process  which  continues  as  a 
program  moves  through  its  lifecycle. 

The  Air  Force  Cost  Analysis  Handbook  (AFCAH)  and  a  GAO  report  of  2009 
outline  several  ways  to  generate  cost  estimates.  From  the  GAO  report,  there  are  three 
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techniques  commonly  used  by  cost  estimators  to  generate  a  point  estimate:  analogy, 
engineering  build-up,  and  parametric.  The  2009  GAO  report  compares  the  three  most 
common  methods  used  in  cost  estimating  and  is  available  in  Table  1 : 

Table  1:  Three  Cost-Estimating  Methods  Compared  (GAO,  2009) 


Method 

Strength 

Weakness 

Application 

Analogy 

■  Requires  few  data 

■  Based  on  actual  data 

■  Reasonably  quick 

■  Good  audit  trail 

■  Subjective 
adjustments 

■  Accuracy  depends  on 
similarity  of  items 

■  Difficult  to  assess 
effect  of  design  change 

■  Blind  to  cost  drivers 

■  When  few  data  are 
available 

■  Rough-order-of-magnitude 
estimate 

■  Cross-check 

Engineering 

build-up 

■  Easily  audited 

■  Sensitive  to  labor  rates 

■  Tracks  vendor  quotes 

■  Time  honored 

■  Requires  detailed 
design 

■  Slow  and  laborious 

■  cumbersome 

■  Production  estimating 

■  Software  developments 

■  Negotiations 

Parametric 

■  Reasonably  quick 

■  Encourage  discipline 

■  Good  audit  trail 

■  Objective 

■  Incorporates  real-world 
effects 

■  Lacks  detail 

■  Model  investment 

■  Cultural  barriers 

■  Need  to  understand 
model’s  behavior 

■  Budgetary  estimates 

■  Design-to-cost  trade 
studies 

■  Cross-check 

■  Baseline  estimate 

■  Cost  goal  allocations 

Analogy 

The  Analogy  method  assumes  that  no  matter  how  technologically  advanced  a  new 
weapon  system  is,  every  weapon  system  is  built  upon  the  knowledge  of  a  previous  weapon 
system.  Establishing  links  between  the  new  system  and  a  previous  system  is  essential  for 
the  Analogy  method.  In  order  to  create  a  traceable  and  repeatable  estimate,  analogy 
estimates  should  be  as  objective  as  possible.  To  create  an  objective  estimate,  the  estimator 
must  select  a  similar  weapon  system  for  comparison.  Historical  data  from  a  previous 
weapon  system  allows  the  estimator  to  create  the  new  weapon  system  estimate  by  adding 
complexity  factors  to  account  for  the  enhancements  on  the  new  weapon  system. 
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Analogous  estimates  are  at  the  beginning  of  a  program  when  little  actual  costs  is  available 
to  the  estimator.  Analogy  estimating  has  several  advantages:  it  is  easy  to  use  before  details 
become  available  on  the  new  weapon  system,  it  is  quick  to  develop  and  at  little  cost,  and  it 
is  easy  to  comprehend  as  the  estimate  stems  from  another  weapon  system.  The 
disadvantages  of  analogous  estimates  are  the  direct  link  of  the  estimate  to  another  weapon 
system,  the  factors  used  are  subjective,  and  no  weapon  system  is  a  perfect  match  to 
generate  an  estimate  (GAO,  2009). 


Engineering  build-up 

When  a  detailed  WBS  is  available,  engineering  build-up  cost  estimates  are  useful. 
Build-up  estimates  start  at  the  lowest  WBS  level  where  labor  hours  and  materials  are 
available.  From  this  point,  an  estimate  for  each  WBS  leaf  is  generated  and  the  leaf 
estimates  are  added  together  to  establish  the  total  weapon  system  estimate.  An  example  of 
a  simple  Aircraft  WBS  is  available  in  Figure  1 : 
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Figure  1:  A  Work  Breakdown  Structure  with  Common  Elements  (GAO,  2009) 

Engineering  build  is  a  good  technique  because  the  method  allows  the  estimator  the  ability 
to  determine  if  he  or  she  accounts  for  all  pieces  of  the  weapon  system.  Engineering 
build-up  also  allows  the  estimator  to  see  which  components  are  the  major  cost  drivers  and 
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if  the  cost  driver  is  transferable  to  other  programs.  The  major  disadvantages  are  build-up 
estimates  are  extremely  expensive  and  time  consuming,  and  all  WBS  elements  must  be 
available  to  generate  a  build-up  estimate  (GAO,  2009). 

Parametric 

Parametric  cost  estimating  uses  statistics  to  develop  relationships  between 
historical  weapon  system  actual  costs  and  a  new  weapon  system.  Parametric  estimating 
uses  a  top-down  approach  to  estimating.  Estimators  create  Cost  Estimating  Relationships 
(CERs)  to  predict  future  costs  on  historical  data  relationships.  Examples  of  categories  that 
use  parametric  estimating  are  weight,  power,  and  lines  of  code.  Regression  is  a  common 
method  to  develop  CERs.  Regression  analysis  allows  the  estimator  to  make  statistical 
inferences.  The  most  important  regression  statistics  to  consider  in  parametric  estimating 
are:  R-squared  (R2),  statistical  significance  (P-value),  F  Statistic,  and  t  Statistic  (GAO, 
2009). 

Additional  Estimating  Techniques 

The  DoD  uses  three  additional  cost  estimating  techniques:  expert  opinion, 
extrapolation  from  actuals,  and  learning  curves.  When  no  data  are  available,  expert 
opinion  is  a  useful  estimating  technique.  Expert  opinions  are  subjective  estimates  which 
provide  a  base  for  generating  a  cost  estimate  (GAO,  2009).  To  establish  a  credible  expert 
opinion  estimate,  the  cost  estimator  must  solicit  information  only  from  the  Subject  Matter 
Expert’s  (SME’s)  field  of  study.  Soliciting  for  point  or  range  estimates  from  SMEs  is  often 
a  difficult  process.  Frequently,  experts  are  reluctant  to  give  a  point  estimate  and  would 
prefer  to  submit  subjective  probability  assessments.  “Subjective  probabilities  are 
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associated  with  one-time,  non-repeatable,  events  whose  probabilities  cannot  be  objectively 
determined  from  a  sample  space  of  outcomes  developed  by  repeated  trials  or 
experimentation”  (Garvey,  2000).  An  example  of  a  subjective  probability  is,  “there  is  a 
50%  chance  the  airplane  will  exceed  $  1M  to  develop.”  This  statement  also  states  there  is  a 
50%  chance  the  airplane  will  not  exceed  $1M.  There  are  always  two  sides  to  subjective 
probability  assessments. 

Extrapolation  from  actuals  is  a  method  used  to  estimate  future  costs  based  on  actual 
costs  or  current  costs.  Averages,  learning  curves,  and  estimates  at  complete  are  examples 
of  extrapolation  techniques.  In  order  to  use  extrapolation  an  estimator  needs  reliable  data 
in  the  correct  fonnat  (labor  hours,  material  dollars,  total  cost)  for  the  estimate. 
Extrapolation  is  best  for  follow  on  work  or  addition  quantities  of  a  weapon  system  where 
the  estimator  knows  the  exact  costs  to  produce  a  weapon  system  (GAO,  2009). 

Learning  curves  allow  an  estimator  to  generate  a  cost  estimate  based  on  the 
knowledge  that  organizations  and  the  people  involved  work  more  efficiently  the  more  they 
perform  a  task.  The  time  to  produce  each  aircraft  should  improve  with  each  model  built. 
GAO  (2009)  states  the  most  common  learning  curve  formula  is: 

Y  =  AXb  (1) 

The  learning  curve  formula  states,  “as  the  number  of  units  doubles,  the  cost  decreases  by  a 
constant  percent”  (GAO,  2009).  Figure  2  shows  a  visual  explanation  of  the  learning  curve. 
Initially,  the  program  experiences  significant  learning,  but  as  the  number  of  units 
developed  increases  the  learning  curve  flattens. 
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Figure  2:  Learning  Curve  Slope  (GAO,  2009) 


Estimating  Cost  Growth 

There  are  two  common  methods  for  calculating  cost  growth.  The  first  method  (2)  is 
to  calculate  cost  growth  as  a  percentage  of  the  original  cost  estimate.  In  the  first  method, 
the  estimated  cost  is  subtracted  from  the  actual  cost  and  then  divided  by  the  estimated  cost 
(McNichols  and  McKinney,  1981). 

( Actual  -  Estimated ) 

Estimated 


The  second  method  (3)  is  to  calculate  cost  growth  as  a  cost  growth  factor  (CGF).  The  CGF 
method  divides  the  estimate  plus  the  cost  variance  (actual)  by  the  estimate  (Dresner  et  ah, 
1993). 
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A  CGF  of  1 .0  indicates  the  program  did  not  go  over  or  under  the  cost  estimate,  and 
the  actual  cost  matched  the  estimated  cost.  If  the  CGF  is  greater  than  1 .0,  the  program 
sustained  growth.  Conversely,  if  the  CGF  is  less  than  1 .0,  the  program  did  not  sustain  cost 
growth;  rather,  the  program  cost  less  than  the  estimate.  To  calculate  the  percent  cost 
growth  subtract  1  from  the  cost  growth  factor  (Dresner  et  ah,  1993). 

Additionally,  Arena  et  al.  (2006)  define  cost  growth  as  the  increase  in  actual  costs 
from  the  most  resent  cost  estimate.  With  multiple  cost  estimates,  CGFs  can  level  out  or 
change  through  time  as  the  cost  estimator  implements  new  estimates  for  a  weapon  system. 
Arena  et  al.  (2006)  report  that  most  previous  studies  discovered  actual  cost  is  greater  than 
the  estimated  or  baseline  cost. 

Three  estimates  exist  within  SARs:  Planning  Estimate  (PE),  Development  Estimate 
(DE),  and  Current  Estimate  (CE)  (Calcutt,  1993).  PEs  are  the  DoD  estimate  made  during 
the  Concept  Exploration  and  Definition  stage  of  the  program  lifecycle.  DEs  estimates 
occur  at  Milestone  (MS)  B  or  the  start  of  EMD  phase  of  the  program  lifecycle.  CEs  are  the 
most  up  to  date  estimate.  If  a  program  is  complete,  the  CE  is  the  actual  cost  of  the  program 
(Calcutt,  1993). 

Estimators  calculate  cost  growth  from  a  baseline  estimate,  the  PE,  DE,  or  CE. 
Typically,  the  DE  at  MS  B  is  the  baseline  estimate  for  cost  growth.  MS  B  is  the  point  in  the 
schedule  where  a  program  enters  full-scale  development  and  officially  becomes  a 
“program  of  record.”  Once  a  program  of  record  is  established,  the  program  is  required  to 
file  official  cost  reports  with  Congress  (Porter  et  al.,  2009).  As  formal  cost  reports 
materialize,  cost  growth  becomes  easier  to  track,  and  it  is  for  this  reason  the  estimator 
measures  cost  growth  from  the  DE  when  possible. 
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Standardization 


Standardizing  certain  variables  is  necessary  to  compare  programs  evenly.  The  two 
variables  with  the  biggest  effect  on  cost  growth  are  inflation  and  order  quantity  (Drezner  et 
ah,  1993).  The  standard  approach  to  account  for  inflation  is  to  convert  all  dollars  to  a 
single  base-year  value.  According  to  Rusnock  (2008)  RAND  prefers  to  adjust  all  program 
dollar  values  to  values  in  the  base-year,  first  year,  of  the  estimate.  A  second  approach  is  to 
adjust  program  dollar  values  to  the  last  year,  or  current  year,  and  make  that  year  the 
base-year.  Because  this  research  includes  some  programs  that  are  not  complete,  we  use 
RAND’s  method  and  establish  the  base  year  as  the  first  year  of  the  estimate  (Rusnock, 
2008). 

When  establishing  the  DE,  the  weapon  system  has  a  planned  number  of  units  to 
procure.  The  estimator  does  not  factor  potential  quantity  changes  into  the  weapon  system 
estimate.  To  combat  quantity  changes,  estimators  generally  use  one  of  two  methods.  The 
first  approach  is  to  adjust  the  CE  to  reflect  DE  quantities.  The  second  approach  is  to  adjust 
the  DE  to  reflect  the  CE.  Rusnock  (2008)  describe  three  methods  Hough  (1992)  outlines  to 
adjust  for  quantity. 

1 .  Standardize  using  variance  listed  in  the  SAR  Quantity  category  only 

2.  Standardize  using  cost-quantity  curves,  thus  adjusting  all  variances  that  occur 
at  other  than  baseline  quantities,  or 

3.  Standardize  using  a  hybrid  approach  by  adjusting  for  quantity-related  variances 
(both  those  listed  in  SAR  Quantity  category  as  well  as  those  listed  in  other 
categories  but  described  as  quantity-related  in  the  narrative  portion  of  the 
SAR)  and  then  adjusting  the  remaining  variance  using  cost-quantity  curves. 
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Arena,  et  al.  (2006)  state  RAND  adopted  the  second  method  described  by  Hough  (1992). 
Additionally,  RAND  standardizes  to  the  final  quantity  and  not  the  quantity  at  the  baseline 
estimate.  Arena  et  al.  (2006)  identify  the  two  major  advantages  of  this  method. 

1 .  The  actual  cost  is  not  changed.  All  estimates  are  adjusted  to  the  actual  cost  and 
the  actual  cost  is  not  changed. 

2.  If  the  other  method  were  used,  adjusting  final  procurement  cost  to  baseline  cost, 
the  total  CGF  would  be  weighted  more  strongly  by  procurement  cost  growth 
(Arena  et  al.,  2006) 

In  addition  to  standardizing  inflation  and  quantity.  Milestone  (MS)  notion  is 
standardized.  In  2000,  MS  notation  shifted  to  MS  A,  B,  C  from  MS,  I,  II,  III.  With  the 
shift  in  notation,  discussions  arose  about  whether  the  notations  are  equivalent  or  not. 
Kassing  et  al.  (2007)  states  MS  II  and  MS  B  as  well  as  MS  III  and  MS  C  are  equivalent. 
Below  is  the  justification  from  Kassing  et  al.  (2007)  for  establishing  MS  B  and  C 
equivalent  to  MS  I,  II: 

We  use  the  current  acquisition  terminology  set  forth  in  DoD  Directive  5000. 1,  May 
12,  2003,  throughout  this  document.  In  accordance  with  this  terminology, 
Milestone  B,  as  of  2000,  represents  the  start  of  the  system  development  and 
demonstration  (SDD)  phase  of  the  DoD  system  acquisition  process.  It  is  defined 
somewhat  differently  than  the  Milestone  II  that  was  used  before  2000,  which  was 
considered  to  be  the  start  of  the  engineering  and  manufacturing  development 
(EMD)  phase.  For  our  analyses,  we  treated  these  two  milestones  as  comparable,  so 
Milestone  B  is  used  throughout  this  monograph  to  mean  Milestone  B  or  an  earlier 
equivalent.  Similarly,  Milestone  C,  the  current  designation  for  the  start  of  the 
production  phase,  is  used  to  mean  Milestone  C  or  its  earlier  equivalent,  Milestone 
IIIA,  which  was  the  authorization  to  start  low-rate  initial  production.  (Kassing  et  al, 
2007) 

Lumb  (2004)  presented  DoD  Business  Transformation,  which  outlines  the 
differences  between  MS  I,  II,  III  and  MS  A,  B,  C  notation.  Figure  3  is  a  visual  description 
of  the  differences  between  the  1996  notation  and  2003  notation. 
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Figure  3:  Defense  Acquisition  Milestones  and  Phases  1996  vs.  2003  (Lumb,  2004) 


Cost  Overrun 

There  are  several  ways  to  measure  cost  overrun.  One  method  is  to  exam  Earned 
Value  Management  (EVM)  data  and  determines  the  difference  between  the  original 
budgeted  amount  and  the  estimate  at  completion.  A  second  method  is  to  examine  the  cost 
growth  factors  which  determine  the  amount  of  cost  growth  at  a  given  point  along  a  weapon 
system’s  schedule.  Christensen  (1994)  used  the  first  EVM  Reports  in  his  analysis  of  cost 
overrun  in  DoD  weapon  systems,  and  states  as  early  as  10%  of  program  completion  cost 
over  begins  to  exist.  Along  with  identifying  cost  overruns  early  in  the  program,  nearly  all 
programs  see  cost  overruns  and  never  recover  after  the  initial  overrun.  Examining  aircraft 
specific  programs,  Christensen  (1994)  discovered  about  75%  of  cost  overrun  occurs  at 
50%  program  completion.  Christensen’s  (1994)  research  provides  a  starting  ground  for 
this  research  in  determining  how  cost  growth  reacts  based  on  aircraft  program  reviews. 
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Cost  Growth  Studies 


This  section  of  Chapter  2  identifies  historical  studies,  which  established  methods 
for  predicting  cost  growth  in  DoD  weapon  systems.  The  studies  include  a  variety  of 
different  methods  to  predict  cost  growth  and  some  of  the  significant  predictors  each  study 
discovered.  From  the  research  to  follow,  this  research  utilizes  some  of  the  predictor 
variables  to  include  in  the  methodology  and  analysis  section.  Several  of  the  studies  come 
from  the  RAND  Corporation  and  others  come  from  research  interested  in  predicting  cost 
growth. 

RAND  and  IDA  Studies 

Asher  et  al.  (1980)  developed  a  method  to  predict  weapon  system  cost  growth. 
SARs  provided  the  data  necessary  to  perform  the  analysis.  Asher  et  al.  (1980)  divided  the 
weapon  system  database  into  different  categories  according  to  the  type  of  weapon  system 
(aircraft,  missile,  ships,  and  other  systems)  and  identified  individual  cost  growth  factors  for 
each  weapon  system.  They  also  developed  a  six-step  approach  to  detennine  development 
and  procurement  cost  growth.  Their  methodology  allows  for  estimator  interpretation  and 
subjective  evaluation  of  the  data.  With  subjective  estimates,  there  is  little  mathematical 
backing  to  support  the  estimates.  In  conclusion,  Asher  et  al.  (1980)  state  cost  estimating 
will  improve  as  the  DoD  program  database  grows  with  future  historical  programs. 

Dresner  et  al.  (1993)  studied  128  weapon  systems  with  development  estimates. 
Their  research  studied  CGFs  of  weapon  systems  during  development,  procurement,  and 
total  program  duration.  Dresner  et  al.  (1993)  found  two  main  factors  affect  cost  growth: 
inflation  and  quantity.  Because  inflation  and  quantity  have  such  a  dramatic  effect  on  cost 
growth,  they  removed  them  from  the  study.  With  the  two  factors  accounted  for,  cost 
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growth  increases  individual  weapon  system  cost  on  average  2.2%  per  year  or  about  20% 
through  the  life  a  program.  Dresner  et  al.  (1993)  discovered  development  CGFs  were  7% 
greater  than  procurement  CGFs.  Another  finding  of  Drezner  et  al  (1993)  is  modification 
programs  sustain  less  cost  growth  than  new  start  programs.  Modifications,  which  cost  less 
than  new  starts,  may  be  an  assumed  finding  as  most  developers  expect  an  estimate  for  a 
weapon  system  modification  to  consist  of  prior  knowledge  that  assists  in  establishing 
estimates  which  are  more  accurate  compared  to  new  developments.  Drezner  et  al.  (1993) 
discovered  longer  program  duration  correlates  to  significantly  greater  cost  growth. 
Program  duration  was  the  only  schedule  variable  that  significantly  correlated  with  cost 
growth. 

The  research  of  Arena  et  al.  (2006)  provides  valuable  information  on  CGFs  for  68 
completed  programs  with  similar  complexities  to  programs  acquired  by  the  U.S.  Air  Force. 
Arena  et  al.  (2006)  identified  three  major  categories  affecting  cost  growth:  acquisition 
strategy,  schedule  factors,  and  other  factors.  Acquisition  strategies  include  but  are  not 
limited  to  prototyping,  modifications,  competition  in  production,  and  contract  incentives. 
Schedule  factors  that  affect  cost  growth  are  program  duration  and  schedule  slip.  Other 
factors  to  consider  when  assessing  the  cause  of  weapon  system  cost  growth  are  poor  cost 
estimates,  and  program  management  decisions  (Arena  et  al.,  2006). 

The  data  used  by  Arena  et  al.  (2006)  came  from  SAR  reports.  The  DoD’s  largest 
weapon  systems,  MDAPs,  are  required  to  annually  file  SARs  with  Congress.  SARs 
provide  a  consistent  platform  for  programs  to  report  financial  data.  Arena  et  al.  (2006) 
only  used  completed  weapon  systems  in  their  study.  They  defined  completed  weapon 
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systems  as  systems  that  have  greater  than  90%  production  complete.  By  using  completed 
weapon  systems,  Arena  et  al.  (2006)  assure  their  analysis  included  nothing  but  final  costs. 

The  data  analysis  section  includes  a  segmented  approach  to  modeling  CGFs.  Arena 
et  al.  (2006)  divided  the  data  into  funding  categories,  milestones,  and  commodity  type,  to 
account  for  possible  changes  in  correlation  with  CGFs.  The  funding  categories  focused  on 
development  and  procurement.  The  MS  category  primarily  focuses  on  MS  II,  and  III. 
Commodities  are  split  into  different  categories  which  include  but  are  not  limited  to: 
aircraft,  missile,  and  ship  (Arena,  2006). 

The  major  findings  from  Arena  et  al.  (2006)  include  significant  cost  growth  at  MS 
II  and  MS  III.  Completed  programs  reported  46%  and  16%  respectfully.  The  two  CGFs 
reported  show  cost  growth  bias  decreases  as  a  program  moves  toward  completion  (Arena  et 
al.,  2006).  This  research  is  significant  to  our  research  because  it  presents  evidence  that  cost 
growth  occurs  at  MS  II  and  III. 

Bolton  et  al.  (2008)  examined  35  MDAPs  from  a  SAR  database  and  unlike  most 
studies,  did  not  standardize  quantity  produced.  The  justification  for  not  standardizing  was 
to  produce  a  true  representation  of  “realized”  cost  growth.  Bolton  et  al.  (2008)  created  four 
categories  for  cost  variances:  errors  in  estimating  and  planning,  decisions  by  the 
government,  financial  matters,  and  miscellaneous  sources.  The  results  attribute  total 
development  and  procurement  cost  growth  to  government  decision  making.  The  biggest 
drivers  of  cost  growth  are  quantity  changes  (21%),  requirements  growth  (13%),  and 
schedule  changes  (9%).  Cost  estimating  contributes  to  10%  of  total  program  cost  growth. 
Overall,  Bolton  et  al.  (2008)  recommend  program  managers,  service  leaders,  and  Congress 
find  ways  to  reduce  the  amount  of  changes  to  requirements  and  quantities  produced  to 
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minimize  cost  growth.  Additionally,  improving  cost  estimates  would  yield  a  significant 
reduction  in  cost  growth. 

Leonard  et  al.  (2013)  focused  on  cost  growth  for  the  entire  acquisition  effort,  and 
define  the  entire  acquisition  effort  from  a  program  point  of  commitment,  typically  MS  B, 
to  system  development.  The  point  of  system  development  is  met  when  a  portion  of 
production  units  planned  at  MS  B  are  produced  and  delivered  to  the  customer.  Leonard  et 
al.  (2014)  divide  the  examined  programs  into  two  groups:  programs  at  least  5  years  past 
MS  B  but  less  than  80%  funded,  and  completed  programs  (at  least  80%  funded).  In  their 
results,  three  continuing  space  programs  had  cost  growth  greater  than  one  standard 
deviation  (extreme  cost  growth).  When  adding  the  F-35  to  the  three  space  programs,  this 
group  makes  up  95%  of  cost  growth  for  all  continuing  programs.  Excluding  the  three 
programs,  MDAPs  started  between  2003  and  2011  sustained  minimal  cost  growth.  With 
smaller  programs  experiencing  minimal  cost  growth,  enhanced  scrutiny  is  placed  on  the 
DoD’s  largest  systems.  In  conclusion,  Leonard  et  al.  (2014)  anticipate  four  programs  will 
consume  the  majority  of  MDAP  funding  for  the  next  20  years:  F-35A,  EELV,  KC-46A, 
and  the  Long  Range  Strike  Bomber. 

AFIT  Research 

The  topic  of  cost  growth  generates  significant  research  attention  at  the  Air  Force 
Institute  of  Technology  (AFIT).  This  section  outlines  several  AFIT  studies  after  2002 
where  AFIT  students  developed  methods  to  predict  total  cost  growth  in  DoD  weapon 
systems.  The  difference  between  past  AFIT  research  and  our  current  research  is  our 
researches  focuses  on  cost  growth  at  different  reviews  throughout  the  program  lifecycle 
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whereas  past  research  focuses  solely  on  total  cost  growth.  Table  2  provides  a  list  of  the 
AFIT  studies  from  2002-2015. 


Table  2:  AFIT  Research 


White  et 

aL 

2004 

Using  Logistic  and  Multiple  Regression  to  Estimate  Engineering  Cost 
Risk 

Moore  & 

White 

2005 

A  Regression  Approach  for  Estimating  Procurement  Cost 

Bielecki 

&  White 

2005 

Estimating  Cost  Growth  From  Schedule  Changes:  A  Regression 
Approach 

Lucas 

2004 

Creating  Cost  Growth  Models  for  the  Engineering  and 

Manufacturing  Development  Phase  of  Acquisition  Using  Logistic  and 
Multiple  Regression 

McDaniel 

&  White 

2007 

Predicting  Engineering  and  Schedule  Procurement  Cost  Growth  for 
Major  DoD  Programs 

Genest  & 

White 

2005 

Predicting  RDT&E  Cost  Growth 

Rossetti 

&  White 

2004 

A  Two-Pronged  Approach  to  Estimate  Procurement  Cost  Growth 
in  Maior  DoD  Weapon  Systems 

Monaco 

&  White 

2005 

Extending  Cost  Growth  Estimation  to  Predict  Schedule  Risk 

Cross 

2006 

Data  Analysis  and  its  Impact  on  Predicting  Schedule  and  Cost  Risk 

Foreman 

2007 

Predicting  the  Effect  of  Longitudinal  Variables  on  Cost  and  Schedule 
Performance 

Rusnock 

2008 

Predicting  Cost  and  Schedule  Growth  For  Military  and  Civil  Space 
Systems 

Deneve 

2015 

A  Macro  Stochastic  Approach  to  Improved  Cost  Estimating  for 
Defense  Acquisition  Programs 

Brown  et 

aL 

2015 

Time  Phasing  Aircraft  R&D  Using  the  Weibull  and  Beta 

Distributions 

White  et  al.  (2004)  was  the  first  of  many  studies  to  use  logistic  and  multiple 
regressions  to  predict  cost  growth  in  DoD  weapon  systems.  White  et  al.  (2004)  focused  on 
predicting  cost  growth  during  the  Engineering  and  Manufacturing  (EMD)  phase  of  the 
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acquisition  life  cycle.  Focusing  on  Research  and  Development  (RDT&E)  dollars  and 
limiting  the  study  to  engineering  cost  growth,  a  logistic  regression  model  predicted  70%  of 
the  validation  data  and  identified  schedule  variables  to  have  the  most  predictive  ability 
(White  et  ah,  2004).  Limiting  the  study  to  the  EMD  phase  allows  the  estimator  to  track  all 
cost  growth  changes  back  to  an  actual  change  of  the  end  item.  Any  cost  growth  in  the 
EMD  phase  is  due  to  a  physical  engineering  change  of  the  weapon  system  (White  et  ah, 
2004).  Their  research  used  SARs  as  the  data  source;  and  the  focus  was  on  the  EMD 
portion  of  the  SAR. 

By  using  historical  data  and  regression  analysis,  White  et  al.  (2004)  were  able  to 
use  an  objective  approach  to  estimate  cost  growth.  They  used  a  two-step  approach  to 
measure  cost  growth  as  a  percentage  change  from  the  development  estimate  to  the  final 
estimate.  First,  the  research  used  logistic  regression  to  determine  if  a  program  sustains  cost 
growth.  White  et  al.  (2004)  identified  the  programs  sustaining  cost  growth,  and  then 
analyzed  the  programs  which  sustained  cost  growth  through  multiple  regression  to  create  a 
model  which  predicts  cost  growth  in  weapon  systems.  Through  regression  analysis,  White 
et  al.  (2004)  established  a  seven-variable  cost  growth  model.  The  predictors  include 
funding  variables,  Time  variable,  weapon  classification,  and  length  of  program.  Using  an 
objective  method  to  predict  cost  growth  establishes  a  foundation  for  future  programs  to 
utilize  in  predicting  cost  growth  (White  et  al.,  2004). 

Bielecki  et  al.  (2005)  and  Moore  et  al.  (2005)  built  upon  the  research  of  White  et  al. 
(2004)  and  generated  models  to  predict  cost  growth  in  different  funding  appropriations 
using  logistic  and  multiple  regression.  Bielecki  et  al.  (2005)  generated  a  model  to  predict 
cost  growth  in  the  RDT&E  budget  during  the  EMD  phase  of  the  program  lifecycle,  and 
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Moore  et  al.  (2005)  generated  a  model  to  predict  cost  growth  in  the  procurement  budget 
during  the  EMD  phase.  Their  research  validates  the  research  of  White  et  al.  (2004)  and 
provides  further  detail  into  the  predictive  characteristics  of  program  cost  growth  in 
RDT&E  and  procurement  budget  categories. 

In  2004,  four  AFIT  students  built  on  the  research  of  White  et  al.  (2004),  Bielecki  et 
al.  (2005)  and  Moore  et  al.  (2005).  Lucas  (2004),  McDaniel  et  al,  (2007),  Genest  et  al., 
(2005),  and  Rosetti  et  al.  (2004)  used  logistic  and  multiple  regression  to  further  enhance 
the  ability  to  predict  cost  growth  in  DoD  weapon  systems.  Lucas  (2004)  focused  on 
developing  a  model  to  predict  a  range  of  cost  growth  for  the  combined  RDT&E  and 
procurement  budgets  in  the  EMD  phase.  McDaniel  et  al.,  (2007)  generated  a  model  to 
estimate  cost  risk  early  in  program  development  which  reduces  the  DoD  cost  growth  rate. 
Genest  et  al.,  (2005)  focused  on  the  pre  EMD  phase  of  a  weapon  system  to  predict  cost 
growth.  Rossetti  et  al.  (2004)  generated  a  model  to  predict  procurement  and  support  cost 
during  the  EMD  phase. 

Cross  (2006)  used  logistic  and  multiple  regression  to  examine  cost  and  schedule 
growth  in  DoD  weapon  systems.  He  focused  on  filling  in  gaps  where  data  was  missing 
from  previous  research,  and  validating  the  schedule  growth  research  of  Monaco  et  al. 
(2005)  and  the  cost  growth  research  of  Genest  et  al.  (2005).  With  a  more  complete  data  set, 
Cross  et  al.  (2006)  highly  recommended  using  schedule  growth  model  of  Monaco  et  al. 
(2005),  but  saw  no  advantage  in  the  cost  growth  model  of  Genest  et  al.,  (2005). 

Foreman  (2007)  continue  to  build  on  the  previous  research  of  cost  and  schedule 
growth.  His  research  focused  on  adding  new  data  sources  and  longitudinal  variables  to 
account  for  changes  in  a  program  over  time.  With  the  new  data  and  longitudinal  variables, 
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Foreman  (2007)  generated  two  regression  models  to  predict  schedule  slip  and  three  models 
to  predict  cost  growth. 

Rusnock  (2008)  focused  on  space  systems.  Using  logistic  and  multiple  regression, 
she  generated  a  model  to  assist  cost  estimators  in  predicting  the  likelihood  of  cost  and 
schedule  growth  in  space  systems.  Her  research  focused  on  two  data  sets.  The  first  data  set 
included  21  DoD  space  programs  required  to  submit  SARs.  The  second  data  set  included 
71  NASA  satellite  and  space  development  programs.  The  results  of  the  research  conclude 
that  program  cost  and  physical  program  size  are  predictors  of  cost  growth.  Additionally, 
Rusnock  (2008)  identified  certain  contractors  as  predictors  of  schedule  growth. 

Deneve  et  al.  (2015)  drove  to  identify  a  method  to  create  more  realistic  cost 
estimates.  Through  analyzing  historical  procedures  to  predict  cost  growth,  their  research 
set  out  to  estimate  if  specific  groups  of  weapon  systems  react  differently  to  cost  growth. 
By  grouping  weapons  systems  together,  Deveve  et  al.  (2015)  looked  to  increase  the 
accuracy  of  estimating  cost  growth.  To  create  a  macro-stochastic  estimation,  Deneve  et  al. 
(2015)  identify  categorical  variables  have  strong  relationships  to  CGFs.  The  important 
groups  fall  into  four  categories:  program  type,  iteration,  funding  years,  and  number  of 
services.  In  addition,  the  study  only  includes  programs  with  MS  B  dates  after  1987,  and 
like  many  other  cost  growth  factor  studies,  this  studied  examines  SAR  data.  Deneve  et  al. 
(2015)  conclude  that  the  groupings  help  predict  the  total  cost  from  the  baseline  estimate 
with  the  DoD’s  largest  programs. 

Brown  et  al.  (2015)  focused  their  research  on  the  cumulative  distribution  function 
to  model  development  expenditures.  Their  research  identified  the  amount  of  program 
expenditures  at  50%  program  completion.  After  calculating  program  expenditures,  Brown 
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et  al.  (2015)  used  regression  analysis  to  detennine  which  program  characteristics  best 
predict  distribution  patterns  for  Rayleigh,  Weibull  and  beta  distributions.  The  program 
characteristics  identified  and  considered  in  the  final  regression  models  include:  contract 
award  before  1985,  upgrade  programs,  length  of  development,  and  first  flight  percent 
schedule  (Brown  et  al.,  2015). 

Predictor  Variables 

Researchers  use  many  variables  to  predict  cost  growth.  In  general,  there  are  four 
basic  variable  groups:  categorical  descriptors,  schedule  related,  cost  related,  and 
performance  related  (Drezner  et  al.,  1993).  Categorical  variables  identify  different  groups 
where  cost  growth  may  exist  and  are  useful  because  they  allow  the  researcher  to  divide 
variables  into  different  subsets  for  analysis.  Common  categorical  subsets  include 
prototype,  service,  new  start  or  modification.  Schedule  variables  are  time  related  variables 
that  can  influence  program  performance  and  are  calculated  using  calendar  dates  listed  in 
SARs  for  Milestones  and  other  important  dates  in  a  program  such  as,  First  Flight  and  Initial 
Operational  Capability.  From  milestone  dates,  the  percentage  change  from  the  planned 
date  to  actual  date  is  calculated  to  identify  schedule  changes  as  a  percentage  (Drezner  et  al., 
1993).  Cost  growth  calculations  use  cost  variables,  which  include  cost  growth  factors, 
program  size,  and  funding  appropriation  distribution  (Drezner  et  al.,  1993).  Perfonnance 
variables  assist  in  predicting  cost  growth  as  well.  Performance  variables  are  similar  to  cost 
variables  in  way  they  are  calculated.  Performance  variables  indicate  if  a  program  achieves 
the  perfonnance  goal  or  not  (Drezner  et  al.,  1993). 

Arena  et  al.  (2006)  report  where  to  expect  differences  between  cost  growth  studies 
based  on  predictor  variables.  Three  factors  contribute  to  differences  in  cost  growth: 
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service,  weapon  system  type,  and  time  trends.  Service  identifies  which  branch  of  the  DoD 
a  system  belongs  to:  Army,  Navy,  Air  Force,  Marines,  Coast  Guard.  Weapon  system 
identifies  which  platform  a  system  belongs  to:  aircraft,  electrical  system,  munitions, 
satellite,  and  missiles.  Time  trend  refers  to  the  period,  year  or  decade,  in  which  the  weapon 
system  was  developed  and  procured.  One  significant  finding  states  cost  growth  was 
significantly  higher  before  the  Packard  Initiatives  in  1969  (Arena  et  al,  2006). 

From  studies  using  program  information  from  SARs,  Arena  et  al.  (2006)  identify 
the  three  most  common  factors  affecting  cost  growth. 

1 .  Acquisition  strategies:  prototyping,  modifications,  contracts 

2.  Schedule  factors:  program  duration,  concurrency,  and  schedule  slip 

3.  Other  factors:  increased  system  capabilities,  unrealistic  cost  estimates,  budget 
trends,  and  management  behavior.  (Arena  et  al.,  2006) 

As  mentioned  earlier,  White  et  al.  (2004)  developed  a  method  to  predict  cost  growth  and 

developed  a  long  list  of  predictor  variables  divided  into  four  main  categories:  program  size, 

physical  type  of  program,  management  characteristics,  and  schedule  characteristics. 

Within  the  four  categories  listed,  some  of  the  variables  are  binary  variables  and  some  are 

continuous  variables.  An  example  of  a  binary  variable  is  the  function  variable  aircraft:  1 

for  yes  the  system  is  an  aircraft  and  0  for  no  the  system  is  not  an  aircraft.  An  example  of  a 

continuous  variable  is  program  maturity  (funding  years  complete).  Program  maturity  is  a 

“continuous  variable  which  indicates  the  total  number  of  years  completed  for  which  the 

program  RDT&E  or  procurement  funding  budget”  (White  et  al.,  2004). 

From  2003-2015  a  wealth  of  AFIT  research  followed  on  the  work  of  White  et  al. 
(2004).  The  follow  on  research  used  the  predictor  variable  database  from  White  et  al. 


(2004)  and  added  some  of  their  own  predictor  variables  to  enhance  the  direction  of  their 
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research.  The  AFIT  research  along  with  other  previous  studies  provide  a  starting  point  for 
our  research  to  predict  cost  growth  in  aircraft  using  program  milestones  and  reviews. 
Table  3  list  potential  predictor  variable  categories  for  this  research. 

Table  3:  Predictor  Variables 


Predictor 

Previously  Documented 

Service 

Arena  et  al.  (2006),  Deneve  et  al  (2015), Dresner  et 
al.  (1993),  McNicol  (2004),  White  etal.  (2004) 

Weapon  System  Type 

Dresner  et  al.  (1993),  Tyson  et  al.  (1994),  White  et 
al.  (2004) 

Time  Trends 

Arena  et  al.  (2006),  Brown  et  al.  (2015),  Deneve  et 
a.  (2015),  McNicol  (1994),  White  etal.  (2004) 

Program  characteristic 

Brown  et  al.  (2015),  Dresner  et  al.  (1993),  Tyson  et 
al.  (1994),  White  etal  (2004) 

Contractor 

Arena  et  al.  (2006),  White  et  al.  (2004) 

Program  size 

White  et  al.  (2004) 

schedule 

White  et  al.  (2004) 

Aircraft  type 

Brown  et  al.  (2015) 

Program  Cost 

Brown  et  al.  (2015),  Deneve  et  al.  (2015), Di bbly 
(1998) 

Summary 

The  literature  review  presented  information  on  methods  for  creating  cost  estimates, 
research  on  when  cost  growth  occurs  in  weapon  systems,  significant  research  on  cost 
growth,  the  development  on  logistic  and  multiple  regression  at  AFIT,  and  potential 
variables  to  consider  in  our  research.  The  next  chapter,  methodology,  discusses  how  we 
approach  addressing  the  goals  of  our  research. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  Chapter  3  is  to  describe  the  data  this  research  uses  in  analysis  and 
the  methods  used  to  evaluate  the  data.  First,  we  discuss  the  data  sources  used  for  our 
analysis.  Second,  we  discuss  the  data  collection  and  evaluation  process.  Third,  we  present 
the  method  used  to  standardize  the  data.  Forth,  we  briefly  discuss  the  process  to  analyze 
cost  growth  at  program  reviews.  Lastly,  this  section  covers  the  procedures  to  identify 
predictors  of  cost  growth  in  aircraft  programs. 

Data  Source 

The  first  step  to  predicting  cost  growth  using  program  reviews  and  milestones  is  to 
establish  a  credible  source  of  data.  Significant  cost  growth  studies  in  the  past  used  SARs  to 
gather  data  for  their  analysis.  In  Table  4,  Arena  et  al.  (2006)  outlines  six  studies  ranging 
from  1982  to  2006  that  used  SAR  data  to  measure  cost  growth.  Additionally  the  AFIT 
research  stream  from  2002-2008  used  SARs  to  conduct  logistic  and  multiple  regression. 
More  recently,  Deneve  et  al.  (2014)  and  Brown  et  al.  (2015)  used  SARs  in  their  research 
discussed  in  Chapter  2.  With  the  studies  above,  we  conclude  that  SAR  data  are  a  reputable 
source  to  begin  model  generation.  An  additional  benefit  of  collecting  data  from  SARs  is 
that  Congress  requires  MDAPs  to  update  and  report  annually  (Brown  et  al.,  2015). 
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Table  4:  CGF  studies  (Arena  et  al.,  2006) 


Citation 

Source 

Time 

Period 

Sample 

Tyson,  Nelson,  Omand 

Pal  mer  ( 1989,  Waif  ( 1990) 

SARs  ( last  SAR  form  Program 
or  December  1987) 

1960-1987 

89  weapon  systems 

Tyson,  harmon,  and  Utech 
(1994) 

SARs  (1st  SAR  for  program  of 
December  1992) 

1962-1992 

20tactical  missiles;  7 

tactical  aircraft 

Dresner  etal.  (1993) 

SARs  ( 1  ast  SAR  for  program  or 
December  1990) 

1960-1990 

128  programs  with  DE 

Shaw  (1982) 

Last  SAR  for  program  or 
December  1982) 

1973-1982 

6 intercept  missile 

programs 

Asher  and  Maggelet  (1984) 

Last  SAR  for  program  or 
December  1983 

As  of  1983 

52  systems  with  IOC 

Arena  etal.  (2006) 

Last  SAR  or  December  2003 

1968-2003 

similar  to  those 
acqui  red  by  US  Ai  r  Force 

SARs  include  cost,  schedule  and  programmatic  information  to  evaluate  program 
execution.  The  cost  data  includes  estimates  at  different  phases  of  a  weapon  system’s  life 
cycle.  Typically,  SARs  provide  a  Planning  Estimate  (PE),  Development  Estimate  (DE), 
and  current  or  final  estimate  (CE).  Along  with  program  estimates,  SARs  report  actual 
costs  incurred  by  their  respective  program  (Dresner  et  al.,1993). 

Over  the  years,  several  organizations  developed  databases  based  off  of  information 
in  SARs  (Arena  et  al.  2006).  White  et  al.  (2004)  developed  a  database  at  AFIT  to  assist  in 
the  research  stream  that  followed  his  work  in  logistic  and  multiple  regression.  RAND  has 
complied  a  database  with  SAR  information  on  DoD  programs  as  well.  This  database 
provides  annual  SAR  funding  reports  by  appropriation  as  well  as  calculated  cost  growth 
measures  (Arena,  2006).  For  our  research,  the  RAND  database  proved  to  be  a  valuable 
source  of  SAR  and  cost  data. 

In  addition  to  SARs,  the  Air  Force  Cost  Analysis  Agency  (AFCAA)  database  of 

aircraft  programs  provides  information  on  program  review  dates.  The  database  includes 
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DoD  aircraft  programs  along  with  significant  program  dates:  MSII,  CDR,  First  Flight, 
DT&E  End,  and  IOC.  The  AFC  A  A  database  along  with  SAR  data  from  RAND  provides 
the  framework  for  establishing  the  working  database  for  this  research. 

The  final  source,  Deagel,  provided  several  program  review  dates  to  complete  the 
database.  Deagel  is  a  civilian  database  that  tracks  civilian  and  military  aircraft  data.  If 
SARs  and  the  AFCAA  database  did  not  provide  a  review  date,  we  referenced  Deagel  to 
complete  the  database.  Additionally,  for  the  programs  with  missing  review  dates,  Deagel  ’ s 
review  dates  align  with  expected  dates  based  off  SAR  and  AFCAA  database  data.  The 
dates  selected  from  Deagel  are  available  in  Table  5. 


Table  5:  Deagel  Program  Dates 


Program  Review 

B-1B 

IOC 

C-17 

CDR 

F-15E 

IOC 

T-45 

IOC 

Data  Collection 

This  research  uses  SAR  data  to  analyze  cost  growth  at  program  reviews.  The 
research  focuses  on  aircraft  programs  in  the  DoD.  “Aircraft  programs  are  defined  as  any 
fixed-wing,  manned  aircraft  developed  for  one  or  more  of  the  US  DoD  service  branches” 
(Brown  et  ah,  2015).  Furthermore,  the  analysis  includes  only  Acquisition  Category  1 
(AC AT  1)  aircraft  programs.  ACAT  1  programs  are  the  highest  dollar  value  acquisition 
programs  in  the  DoD.  To  achieve  ACAT  1  designation  a  program  must  exceed  $480M  in 
Fiscal  Year  (FY)  14  dollars  in  Research  and  Development  (R&D)  or  $2.8B  in  FY14  dollars 
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in  procurement  funding  (Acquisition  Categories  (ACAT)  and  Terms,  2014).  Table  6  lists 
the  30  ACAT  1  Aircraft  programs  to  evaluate  in  this  research. 

Table  6:  Aircraft 


A 10 

AV-8B 

Bl-B 

Bl-A 

C17 

E-8JSTARS 

EF-111A 

E-3A  AWACS 

F14 

B-1B  CMUP  Computer  Upgrade 

F15 

B-2RMP 

F15E 

C-5REP 

F16 

E-2D 

F18A/B 

E-6A 

F18E/F 

EA-18G 

F22 

F-35 (CV) 

F35  (CTOL) 

P-8A 

T6 

S-3A 

T45 

F22  Inc  3.2B 

B-1BCMUPJDAM 

E-3  AWACS  RSIP 

Limitations  of  SARs 

SARs  provide  a  standard  repository  to  obtain  required  annual  reports  for  MDAPs. 
However,  there  are  some  limitations  to  using  SAR  data  in  cost  studies.  Arena  et  al.  (2006) 
and  Hough  (1992)  discuss  some  of  the  limitations  of  SAR  data. 

1 .  SAR  data  are  summary  oriented 

2.  Estimates  reported  change  over  time 

3.  Future  costs  reflect  budgeted  values  and  do  not  necessarily  correlate  to  cost 
estimates 

4.  Report  requirements  change  over  time 

5.  Cost  variances  are  allocated  inconsistently  over  time 
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6.  Each  program  creates  their  SAR  and  all  SARs  do  not  necessarily  report  the  same 
data. 

7.  Only  largest  DoD  program  submit  SARs 

8.  Estimating  techniques  used  are  not  reported  with  baseline  and  current  cost 
estimates 

9.  SARs  do  not  report  risk  and  confidence  levels. 

Data  Set 

This  research  focuses  on  multiple  funding  categories  and  multiple  program 
reviews.  The  funding  categories  are  Development  (Dev),  Procurement  (Proc),  and  total 
program  cost.  The  program  reviews  are  Critical  Design  Review  (CDR),  First  Flight  (FF), 
Development  Test  and  Evaluation  End  (DTE),  and  Initial  Operating  Capability  (IOC). 
Aircraft  programs  recorded  the  four  program  reviews  for  nearly  all  programs  in  this  study. 
In  the  initial  research,  we  considered  other  program  review  dates  such  as  Preliminary 
Design  Review  (PDR),  but  the  aircraft  programs  in  the  dataset  returned  PDR  dates  less 
than  50%  of  the  time.  Therefore,  we  decided  to  exclude  PDR  from  our  analysis. 

In  compiling  IOC  dates,  some  discrepancies  emerged  in  identifying  IOC  dates. 
The  reason  is  programs  are  not  required  to  report  IOC  at  a  certain  point  in  the  program’s 
schedule.  Defense  Acquisition  University  (DAU)  (2015)  defines  IOC,  “In  general, 
attained  when  some  units  and/or  organizations  in  the  force  structure  scheduled  to  receive  a 
system  1)  have  received  it  and  2)  have  the  ability  to  employ  and  maintain  it.  The  specifics 
for  any  particular  system  IOC  are  defined  in  that  system's  Capability  Development 
Document  (CDD)  and  Capability  Production  Document  (CPD).”  With  IOC  defined,  IOC 


dates  reported  by  aircraft  programs  in  this  research  are  consistent  with  the  DAU  definition, 
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where  some  aircraft  programs  report  IOC  earlier  in  the  schedule  than  other  aircraft 
programs. 

Lastly,  in  order  to  calculate  the  percent  complete  of  a  program  we  needed  to 
identify  program  completion.  Because  we  used  SARs  to  analyze  program  cost  growth,  we 
use  the  final  reported  SAR  as  program  completion.  The  final  SAR  (LS)  identifies  when  all 
production  is  complete.  The  USD  AT&L  can  consider  tenninating  SARs  when  90%  of 
production  units  are  complete  or  when  a  program  is  no  longer  considered  an  ACAT  1 
program  (AcqNotes,  20 1 5).  Because  it  is  uncertain  if  termination  of  SAR  reports  occurs  at 
90%  completion  or  at  final  production  completion,  we  use  the  anticipated  date  of  the  last 
production  unit  completion  as  the  LS  and  calculate  the  percent  of  completion  based  off  that 
date. 

Data  Standardization 

In  order  to  conduct  the  analysis  we  need  to  standardize  the  aircraft  program  data. 
The  first  step  is  to  account  for  inflation.  OSD-Comptroller  inflation  rates  convert  all 
program  SAR  values  to  constant  year  (CY)  2015-dollar  values.  Standardizing  the  data 
allows  us  to  evaluate  CGFs  at  the  CY15  dollar  values  instead  of  many  fiscal  year  values 
where  the  significance  of  the  CGF  is  unknown.  See  Table  7  for  the  complete  list  of 
programs  and  associated  program  costs. 
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Table  7:  FY15  Program  $  (Millions) 


Program 

Total  $ 

Proc$ 

RDTE  $ 

Program 

Total  $ 

Proc$ 

RDTE  $ 

A 10 

$10,152.47 

$8,682.68 

$1,469.79 

AV-8B 

$17,811.30 

$15,257.80 

$2,553.50 

Bl-B 

$53,005.07 

$47,037.26 

$5,967.81 

Bl-A 

$56,031.66 

$43,356.75 

$12,674.91 

C17 

$50,462.38 

$43,693.26 

$6,769.11 

E-8  JSTARS 

$4,709.00 

$2,270.64 

$2,438.36 

EF-111A 

$1,881.74 

$1,498.79 

$382.96 

E-3A  AWACS 

$12,086.19 

$8,118.44 

$3,967.75 

F14A 

$32,628.16 

$27,680.35 

$4,947.82 

B-1B  CMUP  Comp 

$553.56 

$223.71 

$329.86 

F15 

$35,059.04 

$26,372.22 

$8,686.82 

B-2RMP 

$1,453.49 

$647.37 

$806.12 

F15E 

$72,474.22 

$66,250.06 

$6,224.16 

C-5REP 

$11,461.23 

$9,592.62 

$1,868.61 

F16A/B 

$18,384.15 

$16,183.54 

$2,200.62 

E-2D 

$16,066.89 

$11,964.82 

$4,102.08 

F18A/B 

$33,376.18 

$27,908.10 

$5,468.08 

E-6A 

$3,717.24 

$3,087.41 

$629.83 

F18E/F 

$89,745.97 

$81,751.88 

$7,994.10 

EA-18G 

$9,663.29 

$7,621.59 

$2,041.70 

F22 

$99,381.83 

$72,505.63 

$26,876.19 

F-35  (CV) 

$103,505.75 

$84,190.95 

$19,314.80 

F35  (CTOL) 

$124,116.40 

$104,860.67 

$19,255.72 

P-8A 

$33,664.17 

$25,620.96 

$8,043.21 

T6 

$4,086.23 

$3,640.13 

$446.09 

S-3A 

$15,313.25 

$12,059.91 

$3,253.35 

T45 

$6,746.34 

$5,774.92 

$971.42 

F22  Inc  3.2B 

$1,600.61 

$355.66 

$1,244.95 

B-1BCMUP  JDAM 

$863.88 

$289.64 

$574.24 

E-3  AWACS  RSIP 

$974.81 

$384.78 

$590.04 

SARs  list  the  quantities  estimated  and  produced  for  each  aircraft  program.  The 
quantities  each  aircraft  program  produces  typically  shift  throughout  the  procurement  stages 
of  a  program’s  lifecycle.  In  order  to  standardize  the  units  produced  for  each  aircraft 
program,  the  units  are  standardized  to  the  final  production  amount.  The  method  used  in 
this  study  is  the  same  method  RAND  adopted  in  1998  (Arena,  2006).  The  standardization 
process  uses  learning  curves  (LC)  and  first  unit  cost  (Tl),  which  is  derived  from  annual 
funding  data  provided  in  each  program  SAR  (Arena,  2006).  If  the  quantity  reported  in  the 
baseline  estimate  is  less  than  the  final  quantity,  we  calculate  the  cost  of  units  not  produced 
and  add  that  value  to  the  baseline  estimate.  Likewise,  if  the  final  quantity  produced  is  less 
than  the  baseline  estimate,  we  calculated  the  estimated  cost  of  additional  baseline  units  and 
subtract  that  value  from  the  baseline  estimate  (Arena,  2006).  Formulas  4-6  outline  the 
calculations  for  Unit  1  cost,  quantity  adjustment,  and  calculating  cost  growth  factors. 
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Calculate  Unit  1  cost: 


( 


Cost 

Quantity 


( LN  (Current  LCSlope)  ^ 

Quantity  LN{2) 


(4) 


Preform  Quantity  Adjustment: 

Z/i(Current  LCSlope)  Z/i(Current  LCSlope) 

((CurrcntQty)($Q//r«;/  T\  )(CurrcntQty)  b,(2)  )  -((BaselineQy%%Ci  irrent  T\  )(Bascl  i  ncQty)  In(2) 


Calculate  Cost  Growth  Factor: 

I  +  ^  CostVariance  -  QtyAdjustment . 
BaselineCost 


(6) 


In  equations  4  and  5,  the  current  learning  curve  slope  is  used  to  calculate  Unit  1  cost 
and  perfonn  quantity  adjustments.  The  learning  curve  slope  explains  that  people  and 
organizations  work  more  efficiently  as  more  units  are  produced.  The  theory  states  that  as 
the  number  of  units  doubles,  the  cost  to  produce  decreases  at  a  certain  rate.  This  is  the 
reason  why  learning  curve  slopes  are  important  to  calculating  cost  growth  factors  in 
aircraft  programs.  LN  represents  the  Natural  Logarithm  in  the  equations  4  and  5. 

Data  Tables 

Table  8  displays  the  complete  CGFs  for  the  30  aircraft  weapon  systems  in  the 

analysis.  The  table  outlines  the  appropriation  and  program  review  for  each  CGF.  The 

blank  fields  in  Table  8  are  due  to  (1)  a  program  not  meeting  the  completion  of  a  program 

review  at  the  time  of  this  analysis,  or  the  program  fell  below  a  SAR  reporting  threshold  and 
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no  longer  required  annual  reports  or  (2),  we  were  unable  to  find  a  recorded  date  for  that 
review.  For  example,  the  F-35  has  yet  to  complete  Development  Test  &  Evaluation  and  the 
Bl-A  fell  below  a  reporting  threshold  in  1978  and  was  no  longer  required  to  make  annual 
SAR  reports;  therefore,  the  fields  are  blank  in  Table  8. 

Descriptive  Statistics 

After  calculating  the  cost  growth  factors,  we  calculate  some  descriptive  statistics 
from  the  cost  growth  factors.  The  research  examines  how  many  programs  and  the  percent 
of  programs  which  sustain  cost  growth  at  each  review,  the  mean,  median,  standard 
deviation  and  interquartile  range  of  the  cost  growth  factors  at  each  review,  and  the 
maximum  and  minimum  responses  of  cost  growth  at  each  review.  These  statistics  help 
better  understand  how  cost  growth  factors  behave  over  time.  In  addition,  we  calculate  the 
percent  of  program  completion  at  each  review.  With  the  percent  cost  growth  and  percent 
program  completion  at  each  review,  we  plot  the  points  which  provides  a  visual  tool  to 
analyze  cost  growth  over  time.  Formula  7  displays  the  formula  necessary  to  calculate 
percent  of  total  cost  growth. 

Percent  of  total  Cost  Growth: 

( Re  view- 1) 

(LS- 1)  1 
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Table  8:  Cost  Growth  Factors 


Development 

Procurement 

Total 

Aircraft 

CDR 

FF 

DTE 

IOC 

LS 

CDR 

FF 

DTE 

IOC 

LS 

CDR 

FF 

DTE 

IOC 

LS 

A 10 

1.09 

1.19 

1.18 

1.19 

1.27 

1.03 

1.22 

1.28 

1.22 

1.34 

1.03 

1.22 

1.28 

1.22 

1.33 

Bl-B 

1.05 

1.05 

1.17 

1.16 

1.31 

0.99 

0.98 

0.96 

0.96 

0.98 

1.00 

0.99 

0.99 

0.98 

1.02 

C17 

1.22 

1.36 

1.41 

1.54 

1.81 

1.08 

1.31 

1.29 

1.45 

1.72 

1.05 

1.33 

1.47 

1.47 

1.75 

EF-111A 

0.97 

1.38 

1.48 

2.10 

2.10 

0.92 

1.53 

1.62 

1.62 

1.62 

0.93 

1.48 

1.60 

1.79 

1.79 

F14 

1.32 

1.32 

1.47 

1.48 

1.83 

1.03 

1.03 

1.19 

0.92 

1.18 

1.08 

1.08 

1.24 

1.02 

1.29 

F15 

0.98 

0.98 

1.09 

1.09 

1.37 

1.05 

1.04 

1.32 

1.23 

1.28 

1.03 

1.03 

1.19 

1.19 

1.30 

F15 

1.07 

1.07 

1.09 

1.09 

1.48 

1.00 

1.00 

1.01 

1.01 

1.01 

1.02 

1.03 

1.04 

1.04 

1.34 

F16 

1.00 

1.25 

1.28 

1.31 

2.51 

1.00 

1.12 

1.10 

1.13 

1.08 

1.00 

1.13 

1.12 

1.15 

1.27 

F18 

1.08 

1.11 

1.15 

1.15 

1.36 

1.02 

1.11 

1.33 

1.35 

1.45 

1.03 

1.11 

1.29 

1.31 

1.43 

F18 

0.99 

0.95 

0.98 

0.98 

0.98 

1.00 

1.02 

0.96 

0.95 

1.01 

1.00 

1.01 

0.96 

0.96 

1.01 

F22 

1.12 

1.19 

1.50 

1.47 

1.64 

1.03 

1.10 

1.61 

1.46 

1.62 

1.05 

1.13 

1.58 

1.47 

1.63 

F35  (CTOL) 

1.25 

1.24 

1.53 

1.36 

1.36 

1.82 

1.26 

1.30 

1.69 

T6 

1.02 

1.02 

0.84 

0.86 

0.90 

1.00 

1.00 

1.42 

1.44 

1.47 

0.99 

0.99 

1.13 

1.36 

1.41 

T45 

1.07 

1.09 

1.31 

1.31 

1.53 

1.10 

1.21 

1.50 

1.48 

1.70 

1.10 

1.20 

1.48 

1.48 

1.68 

B-1BCMUPJDAM 

0.85 

0.80 

0.77 

0.77 

0.77 

1.05 

1.09 

1.05 

1.05 

1.02 

0.94 

0.90 

0.88 

0.88 

0.87 

AV-8B 

1.00 

1.01 

1.21 

1.20 

1.30 

1.00 

1.03 

0.98 

0.86 

0.92 

1.00 

1.03 

1.02 

0.91 

0.98 

Bl-A 

0.96 

1.15 

1.10 

1.00 

1.11 

1.21 

0.99 

1.12 

1.20 

E-8J  STARS 

0.98 

1.22 

2.13 

2.12 

2.41 

1.92 

1.92 

1.87 

1.90 

1.86 

1.06 

1.55 

2.01 

2.02 

2.15 

E-3A  AWACS 

1.52 

1.55 

1.49 

1.71 

1.31 

1.33 

1.32 

1.28 

1.38 

1.41 

1.28 

1.43 

B-1BCMUP  Comp 

0.98 

0.97 

1.02 

1.00 

0.95 

1.00 

0.84 

1.16 

1.02 

0.95 

0.99 

1.07 

1.16 

1.08 

1.07 

B-2RMP 

0.88 

0.81 

1.02 

0.93 

0.93 

1.00 

0.99 

1.14 

1.06 

1.04 

0.93 

0.89 

1.07 

0.99 

0.98 

C-5REP 

0.87 

0.97 

1.00 

1.02 

1.04 

1.03 

1.00 

1.22 

1.01 

0.99 

1.00 

1.21 

E-2D 

1.00 

1.06 

1.26 

1.50 

1.00 

1.09 

1.31 

1.27 

1.00 

1.08 

1.30 

1.33 

E-6A 

1.11 

1.12 

1.11 

1.12 

1.12 

0.97 

0.78 

0.81 

0.82 

0.90 

1.02 

0.87 

0.90 

0.91 

0.98 

EA-18G 

1.05 

1.08 

1.04 

1.04 

1.05 

1.04 

1.04 

1.02 

1.05 

1.05 

1.05 

1.05 

F-35  (CV) 

1.24 

1.50 

1.53 

1.36 

1.70 

1.34 

1.63 

1.62 

P-8A 

0.96 

0.99 

1.11 

1.12 

1.00 

0.95 

0.99 

1.01 

1.00 

1.00 

S-3A 

1.10 

1.08 

1.09 

1.00 

1.02 

1.00 

1.06 

1.02 

1.04 

1.02 

1.07 

F22  Inc  3.2B 

E-3  AWACS  RSIP 

1.02 

1.07 

1.07 

1.07 

1.00 

1.41 

1.57 

2.05 

1.01 

1.21 

1.27 

1.45 

146 

Methodology 

Lastly,  we  discuss  the  process  to  identify  predictors  of  cost  growth  at  program 
reviews.  Step  1 :  we  use  logistic  regression  to  identify  which  programs  are  likely  to  sustain 
cost  growth  at  specific  program  reviews.  Step  2:  we  use  the  results  of  the  logistic 
regression  to  identify  which  continuous  variables  are  predictive  of  cost  growth.  Step  3:  we 
convert  any  significant  continuous  variables  to  categorical  variables.  Step  4:  we  analyze 
these  categorical  variables  and  identify  which  are  significant  using  Fisher’s  Exact  Test.  If 
appropriate,  Step  5:  we  use  the  Odds  Ratio  to  calculate  the  odds  of  the  significant 
categorical  variables  occurring. 
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Logistic  Regression 

The  AFIT  research  stream  from  2002-2008  used  logistic  regression  as  the  first  step 
to  identifying  cost  growth  in  weapon  systems.  With  this  research,  we  follow  the  past  of 
using  logistic  regression  to  identifying  the  likelihood  of  a  program  sustaining  cost  growth. 
Logistic  Regression  uses  binary  (0  or  1)  to  identify  if  a  program  sustains  cost  growth  or 
not.  If  a  program  has  a  CGF  greater  than  1,  positive  cost  growth,  the  program  receives  a 
value  of  ‘  1  ’ .  If  a  program  has  a  CGF  less  than  or  equal  to  1 ,  negative  to  no  cost  growth,  the 
program  receives  a  value  of  ‘O’.  In  our  research,  we  do  not  model  the  likelihood  of  no  or 
negative  cost  growth.  The  purpose  of  identifying  programs  with  positive  cost  growth  is  to 
focus  the  attention  of  the  estimator  on  the  “troubled  programs”.  If  an  estimator  can  identify 
predictors  of  positive  cost  growth  in  the  troubled  programs,  they  may  detennine  a  method 
to  cut  down  total  cost  growth  and  eliminate  cost  growth  across  DoD  weapon  systems. 

With  a  small  sample  size  of  aircraft  programs,  we  split  the  predictor  into  two 
categories,  categorical  variables  and  continuous  variables.  Logistic  regression  examines 
the  continuous  variables  in  an  attempt  to  identify  predictors  of  cost  growth.  For  logistic 
regression,  the  program’s  cost  growth  (0/1)  is  the  (y)  dependent  variable,  and  the  (x) 
independent  variable  is  one  of  the  continuous  variables  (or  predictor  variables)  listed  in 
Table  9.  The  research  analyzes  each  independent  variable  at  the  12  combinations  of  dates 
(CDR,  FF  DTE,  IOC)  and  appropriations  (Development,  Procurement,  Total).  To 
determine  if  an  independent  variable  is  significant,  we  examine  the  p-value  associated  with 
each  test.  In  order  to  consider  an  independent  variable  predictive,  the  p-value  must  be  less 
than  0.10  which  is  our  chosen  significance  level.  Ideally,  the  p-value  returned  is  less  than 


37 


0.05,  but  for  this  analysis,  we  include  any  variable  less  than  0.10  since  our  study  is  more 
exploratory  in  nature  than  confirmatory. 

After  identifying  significant  continuous  variables,  we  convert  the  continuous 
variables  into  a  binary  categorical  variable.  To  convert  a  continuous  variable  to  a 
categorical  variable  it  is  necessary  to  identify  a  “break  or  split”  in  the  data.  At  this  break  or 
split,  we  code  categorical  variables  as:  one  side  of  the  split  equals  ‘0’  and  the  other  side 
equals  ‘1’.  Converting  the  continuous  variables  to  categorical  variables  is  important  for 
this  study  because  of  the  small  sample  size  of  aircraft. 

Table  9:  Continuous  Variables  used  in  Logistic  Regression  Analysis 


Continuous  Variables 
%  RDTE  Funding  @  MSB 
Estimate  MSB-IOC 
Proc  Qty/Months 
MSB  Cost/  Aircraft 
Proc  QTY  @  MSB 
Total  $  (5)  MSB 
Proc  $  @  MSB 
RDTE  $  @  MSB 
Months  from  MSA-MSB 
Months  from  MSB-CDR 
Months  from  MSB-FF 
Months  from  MSB-DTE 
Months  from  MSB-IOC 
Months  from  FF-DTE 


Fisher’s  Exact  Test 

Medical  studies  often  use  Fisher’s  Exact  Test  to  determine  if  (x)  variables  are 
predictive  in  determining  medical  results  or  (y)  variables.  Kennedy  et  al.  (2015)  and 
Pavelites  et  al.  (2014)  use  Fisher’s  Exact  Test  to  determine  if  (x)  variables  are  predictive  of 


medical  outcomes.  Their  research  methods  relate  closely  to  the  goals  of  this  research.  For 
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this  reason,  we  use  Fisher’s  Exact  Test  to  detennine  if  predictors  of  cost  growth,  (x) 
variables,  are  significant  predictors  of  cost  growth  (y  variable)  in  aircraft. 

Due  to  a  small  sample  size,  only  thirty  aircraft  programs,  Fisher’s  Exact  Test  is  an 
appropriate  test  for  independence  between  a  program  having  cost  growth  and  the 
explanatory  variables  considered  in  this  thesis.  Fisher’s  Exact  Test  uses  contingency 
tables,  most  commonly  2*2  tables,  to  test  for  independence.  Fisher’s  Exact  Test  makes 
two  assumptions.  First,  the  test  assumes  all  observations  are  independent.  Second,  the  test 
operates  under  fixed,  or  conditioned,  row  and  column  totals.  The  second  assumption 
distinguishes  Fisher’s  Exact  Test  from  other  statistical  independence  tests  with 
unconditioned  rows  and  columns  (McDonald,  2009).  A  benefit  of  using  Fisher’s  Exact 
Test  is  the  test  does  not  estimate  the  probability  of  a  value;  rather  the  test  calculates  the 
exact  probability  of  receiving  the  observed  data. 

Fisher’s  Exact  Test  accommodates  both  1 -tailed  or  2-tailed  hypothesis  tests,  but 
this  research  uses  just  1 -tailed  hypothesis  tests  to  identify  if  a  categorical  factors  increases 
the  chances  a  program  will  have  a  CGF  >  1.0.  The  null  hypothesis  states  the  categorical 
variable  does  not  effect  the  CGFs.  For  a  right  tailed  Fisher’s  Exact  Test  the  alternate 
hypothesis  is  that  the  CGF  will  be  more  likely  greater  than  1  for  the  categorical  factor  ‘  1  ’ 
than ‘O’. 

•  H0:  the  factors  are  the  same 

•  Ha:  the  probability  (CGF>1)  is  greater  for  the  factor  =  ‘  1  ’  than  ‘0’ 

As  will  be  shown  in  the  next  chapter,  some  predictor  variables  actually  are  predictive  of  a 
CGF<1.  In  that  case,  we  conduct  a  left  tailed  Fisher’s  Exact  Test  to  confirm  the  findings. 
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In  this  scenario,  the  null  hypothesis  remain  the  same  as  the  right  tailed  test,  but  the  alternate 
hypothesis  changes  to: 

•  Ha:  the  probability  (CGF>1)  is  greater  for  the  factor  =  ‘0’  than  ‘  1  ’ 

From  the  complete  pool  of  predictor  variables,  Fisher’s  Exact  Test  examines  the 
categorical  variables,  which  are  given  in  Table  10,  vs  positive  cost  growth  at  the  12  review 
and  appropriation  combinations.  In  addition  to  the  categorical  variables  in  Table  10,  we 
convert  significant  continuous  variables  to  categorical  variables  and  include  them  in  the 
Fisher’s  Exact  Test  analysis. 

Table  10:  Categorical  Variables  used  in  Logistic  Regression  Analysis 


Categorical  Variables 

Fighter/  Attack 

Cohort  3 

Bomber 

Cohort  4 

ISR 

Fairchild 

Trainer 

Rockwel  1 

Cargo/Tanker 

McDonnel  Douglas 

Ai  r  Force 

Northrop  Grumman 

Ai  rcraft 

General  Dynamics 

Prototype 

Lockheed  IVhrti  n 

Modification 

Boeing 

MSB>1985 

Beech  Ai  rcraft 

%  RDTE  Funding  >50% 

Odds  Ratio 

The  Odds  Ratio  (OR)  is  useful  when  interpreting  the  results  of  contingency  tables. 
Because  Fisher’s  Exact  Test  uses  2*2  contingency  tables,  we  are  able  to  calculate  the 
Odds  Ratio  with  the  same  data  tables.  The  Odds  Ratio  is  the  ratio  of  the  odds  of  an  event 
occurring  in  one  group  to  the  odds  of  the  same  event  occurring  in  another  group.  This 
research  predicts  cost  growth  given  different  (x)  predictor  variables.  To  calculate  the  OR, 
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first  identify  that  an  aircraft  has  (x)  variable  and  then  compute  the  odds  the  same  (x) 
variable  has  cost  growth.  Second  identify  the  aircraft  that  do  not  have  the  (x)  variable  and 
then  calculate  the  odds  the  same  (x)  variable  has  cost  growth.  Finally,  divide  the  odds  for 
step  one  by  the  odds  of  step  two  to  determine  the  odds  ratio.  This  ratio  is  stable  for 
relatively  moderate  to  large  sample  sizes.  Due  to  our  small  sample  size  there  is  a 
possibility  the  OR  will  be  unstable.  We  later  show  this  is  the  case.  Therefore,  we  are 
ultimately  unable  to  use  the  OR  to  make  statistical  inferences. 


Summary 

The  Methodology  chapter  discussed  the  data  sources  used  to  gather  program 
funding  information  for  this  research  as  well  as  the  methods  used  to  standardize  the  data. 
Next,  we  described  how  to  calculate  the  cost  growth  factors.  Finally,  Chapter  3  presented 
the  methods  used  to  analyze  the  data:  descriptive  statistics,  logistic  regression,  Fisher’s 
Exact  Test,  and  Odds  Ratio. 
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IV.  Analysis  and  Results 


Introduction 

Chapter  4  starts  with  a  presentation  of  the  descriptive  statistics  associated  with  each 
aircraft  program  and  presents  graphs  on  how  the  cost  growth  behaves  over  time.  Next,  we 
present  the  results  of  logistic  regression  followed  by  Fisher’s  Exact  Test  and,  if 
appropriate,  the  Odds  Ratios.  Finally,  we  present  a  total  analysis  of  the  results. 


Descriptive  Statistics 

Descriptive  statistics  provides  insight  into  the  CGFs  associated  with  each  review 
and  appropriation.  The  categories  included  in  descriptive  statistics  are  sample  size, 
number  of  programs  with  cost  growth,  percent  of  programs  with  cost  growth,  mean, 
median,  standard  deviation,  interquartile  range,  minimum,  and  maximum.  We  used 
Microsoft  Excel  to  calculate  the  descriptive  statistics  in  the  tables  to  follow. 

Table  11  provides  complete  descriptive  statistics  for  the  aircraft  dataset. 

Table  11:  Descriptive  Statistics 


Category 

N  (Sample 
Size) 

Programs  w/ 
Cost  Growth 

%  Programs  w/ 
Cost  Growth 

Mean 

Median 

Standard 

Deviation 

IQR 

Minimum 

Maximum 

CDR 

28 

14 

50% 

1.04 

1.01 

0.11 

0.10 

0.85 

1.32 

Development 

FF 

29 

22 

76% 

1.12 

1.08 

0.17 

0.21 

0.80 

1.52 

DTE 

26 

22 

85% 

1.21 

1.13 

0.27 

0.26 

0.77 

2.13 

IOC 

28 

23 

82% 

1.26 

1.16 

0.32 

0.42 

0.77 

2.12 

LS 

23 

18 

78% 

1.41 

1.31 

0.46 

0.60 

0.77 

2.51 

N  (Sample 

Programs  w/ 

%  Programs  w/ 

Standard 

Category 

Size) 

Cost  Growth 

Cost  Growth 

Mean 

Median 

Deviation 

IQR 

Minimum 

Maximum 

CDR 

28 

13 

46% 

1.07 

1.00 

0.19 

0.05 

0.92 

1.92 

Procurement 

FF 

29 

22 

76% 

1.15 

1.09 

0.24 

0.21 

0.78 

1.92 

DTE 

26 

20 

77% 

1.22 

1.18 

0.25 

0.32 

0.81 

1.87 

IOC 

28 

21 

75% 

1.26 

1.22 

0.32 

0.45 

0.82 

2.05 

LS 

23 

19 

83% 

1.29 

1.21 

0.33 

0.53 

0.90 

2.06 

N  (Sample 

Programs  w/ 

%  Programs  w/ 

Standard 

Category 

Size) 

Cost  Growth 

Cost  Growth 

Mean 

Median 

Deviation 

IQR 

Minimum 

Maximum 

CDR 

28 

15 

54% 

1.03 

1.01 

0.08 

0.06 

0.93 

1.34 

Total 

FF 

29 

23 

79% 

1.13 

1.08 

0.19 

0.20 

0.87 

1.63 

DTE 

26 

19 

73% 

1.21 

1.15 

0.26 

0.27 

0.88 

2.01 

IOC 

28 

21 

75% 

1.25 

1.20 

0.29 

0.44 

0.88 

2.02 

LS 

23 

19 

83% 

1.32 

1.30 

0.31 

0.40 

0.87 

2.15 
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For  the  histograms  of  CGFs  at  the  12  reviews  and  appropriation  combinations,  see 
Appendix  A. 

In  our  analysis,  we  look  to  identify  cost  growth  factors  of  programs  at  different 
reviews.  Figure  4  through  Figure  6  provide  a  visual  representation  of  the  aircraft  programs 
with  positive  cost  growth  factors  vs  percent  of  program  completion.  Each  of  the  program 
reviews  is  identified  by  a  different  shape  as  presented  in  the  legend  of  the  figures.  Our 
analysis  only  includes  programs  that  sustain  cost  growth  because  our  goal  is  to  identify 
how  much  cost  growth  and  the  percent  of  total  cost  growth  an  aircraft  program  is  likely  to 
sustain  given  the  program  sustains  cost  growth.  We  are  not  concerned  with  programs  that 


do  not  experience  cost  growth  because  those  programs  are  performing  as  expected. 


Figure  4:  Development  Cost  Growth 
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Figure  5:  Procurement  Cost  Growth 


Total  CGFs  by  Percent  Complete 


Figure  6:  Total  Cost  Growth 
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In  addition  to  Figure  4  throughFigure  6,  Table  12  lists  the  descriptive  statistics  from 
MSB  to  each  program  review  in  months.  Table  13  identifies  the  mean  and  median  CGF  at 
each  aircraft  review.  Table  14  lists  the  percent  complete  at  each  review  and  the  associated 
percent  of  total  cost  growth  at  that  review.  Table  15  lists  the  average  months  complete  at 
each  review  and  the  associated  percent  of  total  cost  growth  at  that  review.  Lastly,  Figure  7 
through  Figure  10  graphically  display  the  information  presented  in  Table  14  and  Table  15. 


Table  12:  Descriptive  Statistics  MSB-Review  (Months) 


MSB- CDR 

MSB-FF 

MSB- DTE 

MSB- IOC 

MSB-LS 

mean 

25.0 

43.6 

81.3 

88.9 

185.8 

median 

17.2 

34.5 

74.1 

78.1 

176.0 

stdev 

19.0 

24.0 

32.3 

36.9 

95.6 

max 

84.2 

117.7 

170.6 

158.3 

452.5 

min 

1.0 

4.0 

39.5 

32.5 

57.9 

Table  13:  CGF  at  Program  Reviews 


Cost  Growth  Factor 

Review 

mean 

median 

DEV  CDR 

DEV  FF 

DEV  DTE 

DEV  IOC 

DEV  LS 

1.56 

PROCCDR 

PROCFF 

PROC  DTE 

PROCIOC 

1.37 

PROC  LS 

1.37 

TOT  CDR 

1.08 

1.05 

TOT  FF 

1.18 

1.12 

TOT  DTE 

1.29 

1.26 

TOT  IOC 

1.35 

1.31 

TOTLS 

1.40 

1.34 
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Table  14:  %  Program  Completion  vs  %  program  CG 


Mean  %  complete  vs  %  mean  tot  CG 

Median  %  complete  vs  %  med  total  CG 

Review 

%  complete 

mean  %  CG 

Review 

%  complete 

med  %  CG 

DEVCDR 

13% 

22% 

DEVCDR 

DEV  FF 

27% 

33% 

DEV  FF 

DEV  DTE 

49% 

47% 

DEV  DTE 

DEV  IOC 

51% 

60% 

DEV  IOC 

47% 

DEV  LS 

100% 

100% 

DEV  LS 

100% 

PROCCDR 

13% 

44% 

PROCCDR 

PROC  FF 

27% 

59% 

PROC  FF 

PROC  DTE 

49% 

83% 

PROC  DTE 

PROC IOC 

51% 

101% 

PROC IOC 

114% 

PROC  LS 

100% 

100% 

PROC  LS 

100% 

TOTCDR 

13% 

19% 

TOTCDR 

12% 

15% 

TOT  FF 

27% 

45% 

TOT  FF 

25% 

35% 

TOT  DTE 

49% 

72% 

TOT  DTE 

44% 

75% 

TOT  IOC 

51% 

86% 

TOT  IOC 

48% 

91% 

TOT  LS 

100% 

100% 

TOT  LS 

100% 

100% 

Table  15:  Months  Complete  vs  %  Program  CG 


Mean  months  vs  %  mean  total  CG 

Median  months  vs  %  median  total  CG 

Review 

months 

mean 

Review 

months 

median 

DEVCDR 

24.1 

22% 

DEVCDR 

17.2 

20% 

DEV  FF 

43.6 

33% 

DEV  FF 

34.5 

32% 

DEV  DTE 

81.3 

47% 

DEV  DTE 

74.1 

41% 

DEV  IOC 

88.9 

60% 

DEV  IOC 

78.1 

47% 

DEV  LS 

185.8 

100% 

DEV  LS 

176.0 

100% 

PROCCDR 

24.1 

44% 

PROCCDR 

17.2 

18% 

PROC  FF 

43.6 

59% 

PROC  FF 

34.5 

39% 

PROC  DTE 

81.3 

83% 

PROC  DTE 

74.1 

107% 

PROC IOC 

88.9 

101% 

PROC IOC 

78.1 

114% 

PROC  LS 

185.8 

100% 

PROC  LS 

176.0 

100% 

TOTCDR 

24.1 

19% 

TOTCDR 

17.2 

15% 

TOT  FF 

43.6 

45% 

TOT  FF 

34.5 

35% 

TOT  DTE 

81.3 

72% 

TOT  DTE 

74.1 

75% 

TOT  IOC 

88.9 

86% 

TOT  IOC 

78.1 

91% 

TOT  LS 

185.8 

100% 

TOT  LS 

176.0 

100% 
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Mean:  months  Complete  vs  %  Total 
Cost  Growth 


Months  from  MSB 


-Development 

-Procurement 

-Total 


Figure  7:  Mean  months  Complete  vs  %  Total  Cost  Growth 


Median  months  Complete  vs  %  Total 
Cost  Growth 


Months  from  MSB 


-Development 

-Procurement 

-Total 


Figure  8:  Median  months  Complete  vs  %  Total  Cost  Growth 
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Mean:  %  Complete  vs  %  Total  Cost 
Growth 


Figure  9:  Mean  %  Complete  vs  %  Total  Cost  Growth 


Median:  %  Complete  vs  %  Total  Cost 
Growth 


Figure  10:  Median  %  Complete  vs  %  Total  Cost  Growth 

In  the  analysis  to  follow,  we  primarily  address  the  median  values  from  Table  12 
through  Table  15.  The  reason  for  analyzing  the  median  is  the  variability  in  the  data  as 
presented  in  Figure  4  through  Figure  6.  The  figures  show  some  of  the  data  points  could  be 
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more  influential  than  other  data  points.  These  extreme  values  have  a  tendency  to  shift  the 
mean  to  those  points.  Influential  data  points,  however,  do  not  have  as  profound  affect  on 
the  median,  and  it  is  for  this  reason  we  focus  our  analysis  on  the  median.  Table  12  and 
Table  13  provide  summary  tables  of  months  from  MSB  and  CGFs  at  the  reviews.  The 
median  CGFs  at  the  last  SAR  (LS)  are:  DEV  =  1.43,  Proc  =  1.28,  Total  =  1.34.  These 
values  are  the  median  CGFs  expected  when  SARs  are  no  longer  required,  or  a  program 
production  concludes.  This  information  also  shows  that  the  median  CGFs  for  development 
are  greater  than  procurement  and  total  CGFs.  Examining  procurement  CGFs,  the  largest 
median  CGFs  occur  at  DTE  and  IOC,  not  at  the  LS.  This  infonnation  leads  to  further 
analysis  that  Table  14  and  Table  15  assist  in  explaining. 

As  stated  previously,  our  analysis  includes  four  reviews  where  we  examine  CGFs. 
The  four  reviews  all  occur  before  50%  schedule  completion.  IOC  is  typically  the  last 
review  (sometimes  DTE  has  a  later  date),  with  median  percent  complete  of  48%.  Because 
IOC  has  the  latest,  in  terms  of  schedule,  median  percent  complete,  we  further  analyze  this 
review.  Table  14  presents  valuable  information  on  CGFs  in  terms  of  percent  of  schedule 
completion.  At  IOC,  the  procurement  median  CGF  is  1 14%  of  total  cost  growth.  This 
states  that  the  median  CGF  at  IOC  is  greater  than  the  median  LS  CGF,  or  the  program  has 
reached  greater  than  its  peak  CGF  at  IOC.  For  total,  91%  of  total  cost  growth  occurs  at 
IOC.  For  development,  a  much  smaller  percent  of  total  cost  growth  occurs  at  IOC,  47%. 
Table  15  identifies  the  median  months  from  MSB  and  the  percent  of  total  cost  growth 
experienced.  The  median  time  from  MSB  to  IOC  is  78. 1  months  or  6.5  years.  Therefore, 
our  analysis  states  that  the  median  percent  of  total  cost  growth  is  91%  at  a  median  time  of 
6.5  years  after  MSB. 


49 


The  information  from  Table  12  through  Table  15  lends  itself  to  the  same  analysis 
for  CDR,  FF,  and  DTE  as  it  did  for  IOC.  At  CDR,  the  percent  of  total  cost  growth  and 
percent  of  program  completion  are  closer  to  the  same  percent  than  those  values  at  IOC.  At 
FF,  the  same  percentages,  percent  of  total  cost  growth  and  percent  of  program  completion, 
begin  to  separate  as  presented  in  Table  14.  AT  FF,  total  percent  of  total  cost  growth  is  35% 
and  total  percent  of  program  completion  is  25%.  For  total,  we  begin  to  see  percent  of  total 
cost  growth  rise  faster  than  percent  of  program  completion.  At  DTE,  total  percent  of  total 
cost  growth  is  75%  and  total  percent  of  program  completion  is  44%.  Again  the  percent  of 
total  cost  growth  is  rising  faster  at  this  point  compared  to  percent  of  program  completion. 
Lastly,  we  turn  back  to  the  information  presented  on  IOC  and  see  91%  of  total  cost  growth 
occurs  at  48%  schedule  completion.  With  this  information,  we  see  a  steep  rise  in  percent  of 
total  cost  growth  between  FF  and  IOC. 

Procurement  percent  of  total  cost  growth  at  CDR  is  18%  and  percent  of  program 
completion  is  12%.  At  FF  the  percent  of  total  cost  growth  is  39%  and  percent  of  program 
completion  is  25%.  At  FF,  the  percent  of  total  cost  growth  begins  to  increase  more  rapidly 
than  percent  of  program  completion.  At  DTE,  percent  of  total  cost  growth  is  107%  and 
percent  of  program  completion  is  44%.  At  IOC,  percent  of  total  cost  growth  is  1 14%  at 
48%  program  completion.  As  seen  in  Figure  7  through  Figure  10,  procurement 
experiences  a  large  increase  in  percent  of  total  cost  growth  around  DTE  and  IOC.  DTE 
and  IOC  occur  at  44%  and  48%  complete  and  it  is  at  this  point  that  a  program  sees  a  CGF 
greater  than  the  CGF  at  program  completion. 

Development  percent  of  total  cost  growth  does  not  behave  the  same  way  as 
procurement  cost  growth.  At  CDR,  percent  of  total  cost  growth  is  20%  at  12%  program 
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completion.  FF  percent  total  cost  growth  is  32%  and  percent  of  program  completion  is 
25%.  With  CDR  and  FF,  the  percent  of  total  cost  growth  compared  to  percent  of  program 
completion  is  not  too  different,  ~7-8%.  At  DTE  the  percent  of  total  cost  growth  is  41%  and 
at  IOC  the  percent  of  total  cost  growth  is  47%.  Both  of  these  percentages  of  total  cost 
growth  are  less  than  percent  of  program  completion  and  far  less  than  the  percent  of  total 
cost  growth  experienced  with  procurement  at  the  same  reviews. 

The  information  presented  on  percent  of  program  total  program  cost  growth 
can  tie  back  to  a  median  CGF  at  each  review.  Table  13  lists  the  mean  and  median  CGFs  by 
appropriation  and  review.  For  example,  the  median  total  CGF  is  1.31  at  IOC  and  1.34  at 
the  LS.  Additional  discussion  on  this  material  is  offered  in  Chapter  5. 

Logistic  Regression 

Logistic  regression  identified  individual  (x)  variables  that  are  predictive  of  (y) 
variables,  or  cost  growth,  at  different  reviews  in  a  weapon  system.  Table  16  indicates  the 
variables  that  are  significant  predictors  of  cost  growth  at  the  associated  appropriation  and 
review.  Columns  that  are  highlight  grey  with  an  ‘X’  indicate  the  variable  is  predictive  at 
the  0. 1  significance  level  (P-value<0. 10).  Columns  that  not  highlighted  with  an  ‘X’ 
indicate  the  variable  is  predictive  at  the  0.05  significance  level  (P-value<  0.05). 
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Table  16:  (x)  Variables  Tested  as  Predictors  of  Cost  Growth 


Logistic  Regression 

DEV  CDR 

DEV  FF 

DEV  DTE 

DVE  IOC 

Proc  CDR 

Proc  FF 

Proc  DTE 

Proc  IOC 

Total  CDR  Total  FF 

Total  DTE  Total  IOC 

Total 

%RDTE  Funding  @  MSB 

X 

X 

X 

3 

ProcQty/Months 

X 

X 

2 

Proc  QTY  @  MSB 

X 

X 

2 

Months  from  MSA-MSB 

X 

1 

Estimated  MSB-IOC(Mos) 

0 

MSII  Cost  Per  Aircraft 

0 

Total  $  @  MSB 

0 

PROC  $@  MSB 

0 

RDT&E  $  @  MSB 

0 

Mos.  From  MSB-CDR 

0 

Mos.  from  MSB-FF 

0 

Mos.  from  FF-DT 

0 

Mos.  from  MSB-DT 

0 

Mos.  from  MSB-IOC 

0 

Total 

4 

1 

0 

1 

2 

0 

0 

0 

0 

0 

0 

0  8  1 

After  determining  which  continuous  variables  are  significant  in  predicting  cost 


growth,  we  convert  the  continuous  variable  into  a  categorical  variable  to  investigate  its 
effect  as  analyzed  in  Fisher’s  Exact  Test.  %  RDTE  Funding  @  MSB  already  exists  as 
categorical  variable,  %  RDTE  funding  >50%.  If  a  program  has  RDTE  funding  greater  than 
50%  of  total  program  funding  at  MSB,  this  variable  equals  ‘  1  ’  and  all  else  equals  ‘O’.  The 
next  two  significant  continuous  variables:  ProcQty/ Months  and  ProcQTY  @  MSB  are 
highly  correlated.  ProcQty /Months  and  Proc  QTY @  MSB  are  0.8684  correlated  as  seen  in 
Figure:  11. 

This  correlation  is  significant  enough  to  prevent  both  variables  from  being  included 
in  a  regression  model  together.  Therefore,  we  include  one  categorical  variable  to  account 
for  both  continuous  variables.  The  new  categorical  variable  is  Proc  Qty  >  450  where 
programs  with  procurement  quantities  greater  than  450  receive  a  ‘  1  ’  and  all  other  programs 
receive  a  ‘O’.  The  last  continuous  variable  to  convert  to  categorical  is  Months  MSA-MSB. 
The  new  categorical  variable  is  Months  MSA-MSB  >=50  where  any  program  with  months 
from  MSA  to  MSB  greater  than  or  equal  to  50  receives  a  ‘  1  ’  and  all  other  programs 
receives  a  ‘O’. 
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When  examining  Proc  Qty  vs  CGF  we  discovered  a  “break  in  the  plotted  points 
around  450.  At  this  point  we  divided  Proc  Qty  into  the  two  groups  stated  earlier  and  tested 
to  see  to  see  if  the  same  “x”  variables  were  predictive  as  the  continuous  variable  Proc  Qty. 
The  results  were  the  same  and  we  conclude  that  Proc  Qty  >  450  is  a  valid  categorical 
variable.  The  same  methodology  was  followed  for  determining  the  categorical  variable 
Months  MSA-MSB  >=  50.  With  all  predictive  continuous  variables,  transitioned  to 
categorical,  we  now  turn  to  Fisher’s  Exact  Test  results. 


Figure:  11:  Correlation  between  ProcQTY  @  MSB  and  ProcQTY/  Months 

Fisher’s  Exact  Test 

Fisher’s  Exact  Test  tests  for  statistical  associations  between  categorical  (x) 
variables  and  a  dependent  (y)  variable.  Table  17  indicates  significant  predictors  of  cost 
growth  at  different  program  reviews  and  appropriations.  Columns  that  are  highlight  grey 
with  an  ‘X’  indicated  the  variable  is  predictive  at  the  0.1  significance  level  (P-value<0.10). 
Columns  that  not  highlighted  with  an  ‘X’  indicate  the  variable  is  predictive  at  the  0.05 
significance  level  (P-value<  0.05). 
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Table  17:  Fisher's  Exact  Test  Results  Identifying  Significant  (x)  Variables 


Combined 

DEV  CDR 

DEV  FF 

DEV  DTE 

DVE  IOC 

Proc  CDR 

Proc  FF 

Proc  DTE 

Proc  IOC 

Total  CDR  Total  FF 

Total  DTE 

Total  IOC 

Proc  Qty  >450 

X 

X 

2 

Months  MSA-MSB  >=  50 

X 

1 

Bomber 

X 

X 

X 

X 

X 

5 

Weapon  Type  Aircraft 

X 

X 

X 

X 

4 

Prototype 

X 

X 

X 

X 

X 

5 

MSB  estimate  >1985 

X 

X 

X 

2 

Modification 

X 

X 

X 

3 

%RDTE  Funding  >50% 

X 

X 

2 

Air  Force 

X 

1 

Fighter/  Attack 

X 

1 

5 

5 

1 

5 

2 

2 

0 

2 

2 

3 

0 

2 

The  p-values  only  indicate  if  the  variable  is  significant  or  not.  In  order  to  determine 


the  significance  of  each  variable  it  is  necessary  to  identify  if  the  results  are  ‘left’  or  ‘right’ 
tailed  test.  For  each  (x)  variable  identified  as  significant  in  Table  17,  Table  18  -  Table  27 
identify  the  appropriation,  review,  tail,  and  probability,  as  well  as  a  short  explanation  of 
what  the  results  indicate. 

Looking  across  the  appropriation  type  and  review  dates,  we  can  see  what  type  and 
how  many  explanatory  variables  are  statistically  predictive.  For  example  (x)  Variables  in 
development  are  significant  16  time,  (x)  variables  in  procurement  are  significant  6  times 
and  (x)  variables  in  total  funding  are  significant  7  times.  This  result  indicates  development 
(x)  variables  are  more  predictive  compared  to  procurement  and  Total. 


Table  18:  the  probability  of  a  CGF  greater  than  1  is  greater  for  programs  with  Proc 

Qty >  450 


X  Variable 

Appn 

Review 

Tail 

Prob 

Proc  Qty  >450 

DEV 

CDR 

Right 

0.0601 

Proc  Qty  >450 

Proc 

CDR 

Right 

0.0601 
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Table  19:  the  probability  of  a  CGF  greater  than  1  is  greater  for  programs  with 


Months  MSA-MSB  >=50 


X  Variable 

Appn 

Review 

Tail 

Prob 

Months  MSA-MSB  >=  50 

Dev 

CDR 

Right 

0.043 

Table  20:  the  probability  of  a  CGF  less  than  1  is  greater  for  Bomber  Aircraft 


X  Variable 

Appn 

Review 

Tail 

Prob 

Bomber 

Dev 

FF 

Left 

0.0747 

Bomber 

Dev 

IOC 

Left 

0.0115 

Bomber 

Total 

CDR 

Left 

0.0131 

Bomber 

Total 

FF 

Left 

0.0456 

Bomber 

Total 

IOC 

Left 

0.0376 

Table  21:  the  probability  of  a  CGF  greater  than  1  is  greater  for  Aircraft  Programs 

than  Electronic  Upgrade  Programs 


X  Variable 

Appn 

Review 

Tail 

Prob 

Weapon  Type  Aircraft 

Dev 

CDR 

Right 

0.0489 

Weapon  Type  Aircraft 

Dev 

FF 

Right 

0.0339 

Weapon  Type  Aircraft 

Dev 

IOC 

Right 

0.0115 

Weapon  Type  Ai rcraft 

Dev 

FF 

Right 

0.0349 

Table  22:  the  probability  of  a  CGF  greater  than  1  is  greater  for  programs  with 

Prototypes 


X  Variable 

Appn 

Review 

Tail 

Prob 

Prototype 

Dev 

FF 

Right 

0.0187 

Prototype 

Dev 

IOC 

Right 

0.0886 

Prototype 

Proc 

FF 

Right 

0.0547 

Prototype 

Proc 

IOC 

Right 

0.0386 

Prototype 

Total 

IOC 

Right 

0.0934 
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Table  23:  the  probability  of  a  CGF  less  than  1  is  greater  for  programs  with  MSB 


estimate  >  1985 


X  Variable 

Appn 

Review 

Tail 

Prob 

MSB  estimate  >  1985 

Dev 

FF 

Left 

0.049 

MSB  estimate  >  1985 

Dev 

DTE 

Left 

0.0478 

MSB  esti  mate  >  1985 

Dev 

IOC 

Left 

0.0306 

Table  24:  the  probability  of  a  CGF  less  than  1  is  greater  for  Modification  Programs 


X  Variable 

Appn 

Review 

Tail 

Prob 

Modification 

Proc 

CDR 

Left 

0.0642 

Modification 

Proc 

CDR 

Left 

0.0642 

Modification 

Total 

FF 

Left 

0.0019 

Table  25:  the  probability  of  a  CGF  less  than  1  is  greater  for  programs  with  %  RDTE 

Funding  >  50% 


X  Variable 

Appn 

Review 

Tail 

Prob 

%  RDTE  Funding  >50% 

Dev 

CDR 

Left 

0.0824 

%  RDTE  Funding  >50% 

Dev 

FF 

Left 

0.0747 

Table  26:  the  probability  of  a  CGF  greater  than  1  is  greater  for  Air  Force  Programs 


X  Variable 

Appn 

Review 

Tail 

Prob 

Air  Force 

Proc 

IOC 

Right 

0.0189 

Table  27:  the  probability  of  a  CGF  greater  than  1  is  greater  for  Fighter/  Attack 

Aircraft 


X  Variable 

Appn 

Review 

Tail 

Prob 

Fighter/  Attack 

Total 

FF 

Right 

0.0169 
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We  intended  to  present  ORs  for  each  Fisher’s  Exact  Test  but  the  small  sample  size 
produced  unstable  results.  As  an  example  the  OR  for  months  MSA -MSB  >=  50  produced 
an  OR  with  an  approximate  95%  Confidence  Interval  for  (1.06,  166.37).  This  large  width 
suggests  instability  of  the  OR. 

For  references  on  the  contingency  tables  used  to  calculate  Fisher’s  Exact  Test  and 
Odds  Ratios,  please  see  the  Appendix.  From  the  results  in  Table  18  through  Table  27,  we 
can  determine  which  direction  is  beneficial  for  programs  to  sustain  less  cost  growth 
throughout  the  program  schedule.  Table  28  indicates  the  cells  coded  “Grey”  are  positive 
and  the  cells  coded  “White”  are  negative  in  terms  of  their  impact  on  a  program  sustaining 
cost  growth. 


Table  28:  Categorical  Variable  Stoplight 


Nominal  Variable 


NO  =  'O'  Yes  =T 


Proc  Qty  >  450 
Months  IVBA-IV6B>=50 
Bomber 

VWapon  Type  Ai  rcraft 
Prototype 

MSB  esti  mate  >1985 
Modification 
%  RDTE  Funding  >50% 
Air  Force 
Fighter/  Attack 


Additionally,  Table  17  identifies  the  how  many  of  the  12  possible  reviews  and 
appropriation  combinations  a  (x)  variable  is  predictive  for.  Table  29  is  a  compressed 
version  of  Table  17  and  lists  the  (x)  variables  from  greatest  to  least  number  of  reviews  and 
appropriations  significant.  The  results  show  Bomber  and  Prototype  have  the  greatest 
results  with  (5/12)  significant  followed  by  Weapon  Type  Aircraft  with  (4/12)  significant. 
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Table  29:  Significant  # 


X  variable 

#  significant 

Bomber 

5 

Prototype 

5 

Weapon  Type  Aircraft 

4 

Modification 

3 

ProcQty  >450 

2 

MSB  estimate  >1985 

2 

%RDTE  Funding  >50% 

2 

Months  MSA- MSB  >=  50 

1 

Air  Force 

1 

Fighter/  Attack 

1 

Overall,  our  analysis  generated  significant  results.  Table  12  through  Table  15  and 
Figure  4  through  Figure  6  display  valuable  information  on  how  aircraft  cost  growth 
behaves  over  the  life  of  a  program.  At  IOC,  total  percent  of  total  cost  growth  is  91%  at 
48%  program  completion.  For  procurement,  1 14%  of  total  program  cost  growth  occurs  at 
48%  program  completion  and  for  development,  only  47%  of  total  program  cost  growth 
occurs  at  48%  program  completion.  Additionally,  for  procurement  and  total,  a  spike  in 
percent  of  total  program  cost  growth  occurs  around  FF  whereas  development  cost  growth 
follows  a  steadier  path. 

The  second  part  of  the  analysis  involving  logistic  regression  and  Fisher’s  Exact 
Test  returned  significant  results  in  predicting  program  cost  growth.  Bomber,  Prototype, 
and  Weapon  type  are  the  most  predictive  variables  of  cost  growth.  Additionally, 
explanatory  variables  are  predictive  in  development  16  times,  procurement  6  times,  and 
total  7  times. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

Chapter  5  revisits  the  initial  investigative  research  questions  to  support  our  analysis 
and  conclusions.  Second,  we  offer  limitations  of  the  research  and  a  brief  discussion  on  one 
of  the  intended  goals  we  were  not  able  to  accomplish.  Lastly,  we  provide  several  ideas  for 
future  research  and  a  summary  of  how  this  research  provides  value  to  the  Air  Force  and 
DoD. 

Research  Goals  Answered 

In  our  research,  we  look  to  identify  a  significant  review  along  an  aircraft’s 
schedule  where  cost  growth  occurs.  After  identifying  cost  growth  factors  at  four  program 
reviews,  we  look  to  identify  predictors  of  cost  growth  at  that  point  in  time.  Lastly,  the 
research  graphically  depicts  the  trends  of  cost  growth  along  an  aircraft’s  schedule. 

1.  Identify  a  significant  review  along  an  Aircraft’s  schedule  where  cost 
growth  occurs. 

Up  to  this  point,  research  examined  at  what  percent  of  program  completion  cost 
overrun  or  cost  growth  begins  (Christensen,  1994).  Christensen  (1994)  identified  cost 
overruns  starting  around  10%  program  completion.  In  our  research,  we  do  identify  CGFs 
at  all  four  reviews  in  the  analysis,  but  our  analysis  provides  more  valuable  infonnation  than 
simply  identifying  where  cost  growth  occurs.  From  Table  14  and  Table  15  we  gain 
important  information  on  how  cost  growth  behaves  depending  on  the  appropriation  and  the 
review  along  an  aircraft  program’s  schedule. 
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As  presented  in  Chapter  4,  the  median  percent  of  program  completion  for  IOC  is 
48%  and  the  median  percent  of  total  cost  growth  for  total  appropriation  is  91%.  Therefore, 
we  identify  the  cost  growth  factor  of  a  program  at  IOC  to  be  very  close  to  the  cost  growth 
factor  at  program  completion.  When  we  examine  total  FF,  we  see  the  first  major  spike  in 
percent  of  total  cost  growth  compared  to  percent  of  program  completion.  At  FF,  the 
median  percent  of  total  cost  growth  is  35%  at  25%  program  completion.  Here  we  see  a 
spike  in  the  rate  of  cost  growth,  which  could  be  attributed  to  a  program  actually  needing  to 
display  some  capability  for  the  aircraft.  FF  is  not  the  first  time  we  see  cost  growth. 

The  percent  of  total  cost  growth  at  CDR  is  15%  where  the  percent  of  program 
completion  is  12%.  At  CDR,  an  aircraft’s  cost  growth  is  growing  at  roughly  the  same 
percentage  of  schedule  completion.  When  looking  at  DTE,  total  percentage  total  cost 
growth  is  75%  at  44%  program  completion.  So,  at  DTE,  there  is  a  major  spike  in  percent  of 
total  cost  growth.  From  Figure  6,  we  see  DTE  does  not  necessarily  occur  before  IOC.  The 
reason  for  this  is  stated  is  Chapter  3,  where  IOC  does  not  necessarily  occur  at  the  same 
point  along  a  program’s  schedule.  Because  of  this,  DTE  can  occur  after  IOC  depending  on 
where  IOC  is  identified  in  a  program’s  CDD.  Due  to  shifts  in  IOC,  the  point  of  greatest 
CGF  could  occur  at  DTE  or  IOC. 

Looking  deeper  into  the  appropriations,  development  and  procurement,  we  see 
much  different  results  for  percent  of  total  cost  growth  vs  percent  of  program  completion. 
For  development,  median  percent  of  total  cost  growth  at  IOC  is  47%  whereas  median 
percent  total  cost  growth  for  procurement  is  1 14%.  With  this  infonnation,  we  are  likely  to 
see  development  cost  growth  after  IOC  but  do  not  expect  to  see  any  procurement  cost 
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growth  after  IOC.  With  this  information,  we  should  focus  our  attention  on  development 
after  IOC  to  minimize  total  program  cost  growth. 

Our  analysis  also  identifies  the  amount  of  time  from  MSB  to  each  program  review. 
The  median  months  from  MSB  to  each  review  is  available  in  Table  12.  The  median  time 
from  MSB  to  IOC  is  78  months  or  6.5  years.  Therefore,  at  a  median  of  6.5  years  after  MSB 
a  program  sustains  about  91%  of  the  total  program  cost  growth.  Additionally,  the  first 
spike  in  percent  of  total  cost  growth  occurs  at  FF,  35  months  or  -3  years. 

All  of  the  information  on  percent  of  total  cost  growth,  percent  of  program 
completion,  and  time  from  MSB  can  tie  to  a  median  CGF  at  the  specific  review.  Table  13 
identifies  the  mean  and  median  CGFs  at  each  review  and  appropriation.  This  table  allows 
us  to  put  a  median  CGF  to  a  program  at  each  review.  With  future  programs,  the 
information  on  median  CGF,  percent  of  total  cost  growth,  and  percent  of  program 
completion  can  all  contribute  to  creating  more  accurate  estimates  as  a  program  moves 
toward  completion. 

2.  Identify  predictors  of  cost  growth  at  program  reviews. 

Using  Fisher’s  Exact  Test,  we  identified  significant  predictors  of  cost  growth  at  the 
four  reviews,  CDR,  FF,  DTE,  and  IOC.  FF  identified  the  most  (x)  variables  as  predictors, 
10.  CDR  and  IOC  both  identified  9  (x)  variables  as  predictors  of  cost  growth.  DTE  End 
only  identified  1  (x)  variable  as  a  predictor  of  cost  growth.  Bomber,  Prototype,  and 
Weapon  Type  Aircraft  displayed  the  most  importance  as  they  were  significant  (5/12), 
(5/12),  and  (4/12)  respectively.  Bomber  aircraft  tend  to  have  less  cost  growth  compared  to 
other  airframes.  If  a  program  has  prototypes,  the  program  is  likely  to  experience  more  cost 
growth  compared  to  programs  without  prototypes.  Significant  results  for  prototypes  agree 
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with  the  past  results  of  Dresner  et  al.  (1993),  White  et  al.  (2004),  and  Tyson  et  al.  (1994) 
where  they  identified  prototyping  as  a  predictor  of  cost  growth.  Lastly,  we  found  aircraft 
are  more  likely  to  experience  cost  growth  compared  to  aircraft  electronic  systems 
upgrades.  Identifying  upgrades  as  a  significant  predictor  agrees  with  Brown  et  al.  (2015) 
where  they  identified  upgrade  characteristics  as  a  predictor  variable  for  their  distribution 
parameters 

3.  Graphically  display  trends  of  cost  growth  along  an  aircraft’s  schedule 
based  on  program  reviews. 

Figure  4  through  Figure  6  display  cost  growth  trends  along  an  aircraft’s  schedule 
for  development,  procurement  and  total  appropriations.  The  major  findings  from  the  cost 
growth  trends  are  presented  earlier  in  this  chapter  under  Research  Goal  1 . 

Limitations 

In  our  analysis  of  cost  growth  in  DoD  aircraft  programs  the  major  limitation  was 
the  lack  of  reported  program  reviews.  We  were  only  able  to  gather  data  for  four  significant 
program  reviews,  CDR,  FF,  DTE  End,  and  IOC.  Ideally,  the  study  would  consider 
additional  dates,  [Preliminary  Design  Review  (PDR),  Operational  Test  and  Evaluation 
(OTE),  Fully  Operational  Capability  (FOC)]  but  the  data  was  not  there  to  support  deeper 
analysis.  SARs  do  not  include  all  program  dates,  and  the  AFCAA  database  did  not  include 
additional  dates.  Compiling  a  database  directly  from  weapon  system  program  offices 
would  allow  for  complete  analysis  and  potentially  generate  additional  significant  results  or 
an  implementable  model. 
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Recommendations  for  Future  Research 


We  recommend  several  areas  for  future  research.  First,  our  study  included  30 
aircraft  programs.  This  research  started  with  a  specific  platform,  but  the  research  could 
expand  to  include  other  platforms:  Space,  Ship,  Land,  Missile,  and  other  DoD  systems. 
Expanding  the  field  of  study  could  provide  valuable  infonnation  on  the  differences 
between  DoD  platforms  in  terms  of  how  the  programs  experience  cost  growth  throughout 
their  lifecycle.  Second,  as  stated  in  the  limitations  section,  our  study  used  program  reviews 
in  analyzing  cost  growth.  Collecting  program  office  data  could  allow  for  the  inclusion  of 
additional  dates  in  the  analysis.  Lastly,  we  recommend  a  longitudinal  study  of  cost  growth. 
Foreman  (2007)  conducted  a  longitudinal  study  to  identify  how  a  program  behaved  over 
time.  His  study  along  with  the  dataset  and  results  from  our  study  provide  a  starting  ground 
to  analyze  how  programs  behave  overtime.  Some  potential  research  questions  for 
longitudinal  studies  are:  If  a  programs  sustains  x%  cost  growth  at  CDR,  what  does  the  cost 
growth  %  look  like  at  subsequent  reviews?  Will  cost  growth  continue  or  is  there  any 
chance  for  recovery?  Is  there  a  point  where  the  program  can  expect  extreme  cost  growth 
(greater  than  50%)?  We  believe  both  future  research  topics  provide  paths  to  discovering 
significant  results  to  combat  cost  growth  in  the  DoD. 

Last  Words 

Our  research  identified  CGFs  and  percent  of  total  cost  growth  at  program  reviews. 
In  past  research,  we  do  not  find  any  research  which  directly  validates  our  findings  using 
CGFs  at  different  reviews  along  a  program’s  schedule.  However,  there  is  significant 
research  on  total  CGFs  in  different  appropriations.  Arena  et  al.,  (2006)  found  total  CGFs 
for  development,  procurement,  and  total  to  be  1.58,  1.44,  and  1.46  respectively  and 
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Dresner  et  al,  (1993)  found  total  CGFs  for  development,  procurement  and  total  to  be  1 .25, 
1.18,  and  1.20  respectively.  Additionally,  Christensen,  (1993),  uses  EVM  data  to  identify 
cost  overrun  beginning  as  early  as  10%  of  program  completion.  Our  research  identifies 
total  cost  growth  at  CDR,  which  occurs  at  12%  program  completion.  Our  research  ties  to 
the  finding  of  Christensen  (1993).  Brown  et  al.,  (2015)  presented  several  models  to  predict 
the  percent  of  expenditures  occurring  at  50%  program  completion.  Our  findings,  along 
with  the  findings  of  Christensen  (1993),  Dresner  et  al.,  (1993),  Arena  et  al.  (2006),  and 
Brown  et  al.  (2015),  contribute  to  creating  more  accurate  cost  models  in  the  future. 


Appendix  A:  CGFs  vs  Probability  of  Occurrence 
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Figure  A2:  Development  FF 


Figure  A3:  Development  DTE 


Figure  A4:  Development  IOC 
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Figure  A5:  Procurement  CDR 


Figure  A6:  Procurement  FF 
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Figure  A7:  Procurement  DTE 


Figure  A8:  Procurement  IOC 


Figure  A9:  Total  CDR 
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Figure  A10:  Total  FF 


Figure  All:  Total  DTE 


Figure  A12:  Total  IOC 
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Appendix  B:  Contingency  Tables 
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Figure  A13:  DEV  FF  >  1  vs  Bomber 
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Figure  A14:  Dev  IOC  >  1  vs  Bomber 
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A  Mosaic  Plot 
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Figure  A15:  Total  CDR  >  1  vs  Bomber 
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Figure  A16:  Total  FF  >  1  vs  Bomber 
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Figure  A17:  Total  IOC  >  1  vs  Bomber 
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Figure  A18:  Dev  FF  >  1  vs  Prototype 
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Figure  A19:  Dev  IOC  vs  Prototype 
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Figure  A20:  Proc  FF  >  1  vs  Prototype 
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Figure  A21:  Proc  IOC  >  1  vs  Prototype 
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Figure  A22:  Total  IOC  >  1  vs  Prototype 
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Figure  A23:  Dev  CDR  >  1  vs  Weapon  Type  Aircraft 
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Figure  A24:  Dev  FF  >  1  vs  Weapon  Type  Aircraft 
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Figure  A25:  Dev  IOC  >  1  vs  Weapon  Type  Aircraft 
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Figure  A26:  Total  DCR  >  1  vs  Weapon  Type  Aircraft 
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Figure  A27:  Proc  CDR  >  1  vs  Modification 
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Figure  A28:  Proc  FF  >  1  vs  Modification 
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Figure  A29:  Total  FF  >  1  vs  Modification 
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Figure  A30:  Dev  CGF  >  1  vs  Proc  QTY  >  450 
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Figure  A  31:  Proc  CDR  >  1  vs  Proc  Qty  >  450 
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Figure  A32:  Dev  FF  >  1  vs  MSB  estimate  >  1985 
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Figure  A33:  Dev  DTE  >  1  vs  MSB  estimate  >  1985 
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Figure  A  34:  Dev  IOC  >  1  vs  MSB  estimate  >  1985 
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Figure  A35:  Dev  CDR  >  1  vs  %  RDTE  funding  >  50% 
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Figure  A36:  Dev  FF  >  1  vs  %  RDTE  funding  >  50% 
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Figure  A37:  Dev  CDR  >  1  vs  Months  from  MSA  -  MSB  >=  50 
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Figure  A38:  Proc  IOC  >  1  vs  Air  Force 
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Figure  A39:  Total  FF  >  1  vs  Figher/  Attack 
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Appendix  C:  Variables  defined 


Variable  defination 

Cohort  3  Procurement  Quantity  at  MSB  >=  1000 
Total  estimated  cost  at  MSB  >=  $100000 
Cohort  4  Total  esti mate  at  MSB  >$99000 
Months  from  MSA  to  MSB  >55 
Prototype  =  Yes 
Modification  =  No 


Figure  A40:  Cohorts  Defined 
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